R 4.2 for Windows will support UTF-8 as native encoding, which will be amajor improvement in encoding support, allowing Windows R users to workwith international text and data.
This new feature will require at least Windows 10 (version 1903) ondesktop systems, Windows Server 2022 on long-term support server systemsor Windows Server 1903 from the semi-annual channel. Older Windows systemswill be able to run R, but with the same limitations in the encoding supportas in R 4.1 and earlier.
As part of this change, R will require UCRT as the new C runtime forWindows. This means that on desktop systems older than Windows 10 and onserver systems older than Windows Server 2016,UCRTwill have to be installed before installing R. MSVCRT, the older C runtime,will no longer be supported. R 4.2 will also drop support for 32-bit buildson Windows.
A new compiler toolchain, Rtools42, will be used for building Windowsbinaries of R and R packages from source. All code will have to be rebuiltwith the new toolchain.
Nothing will change for R 4.1.x, not even for the upcoming minor revisions.They will still use the current encoding support and be built using Rtools4in 32-bit and 64-bit versions for MSVCRT.
Nothing will change for end R-users before R 4.2, they don’t have to doanything special now. Only users of R-devel on Windows will be affectedbefore the release.
The change will, however, require some cooperation from some packageauthors. Most authors will not have to do anything as the number of CRANpackages that will need some attention is below 1%, but authors of packagesusing native (C, C++ or Fortran) code should read the following lines. Someof them will have to update their packages, but in most cases they may usepatches created by Tomas Kalibera and/or receive more advice from TomasKalibera or Uwe Ligges. We thank the package authors who already have beenworking with us on the change for their cooperation.
Current state
So far, R-devel snapshot binary builds and binary builds of R packages onCRAN have been built using Rtools4 (GCC 8) and used MSVCRT as the C runtime.Thanks to Jeroen Ooms for putting together and maintaining Rtools4 and thebinary builds of R. MSVCRT does not allow using UTF-8 as native encoding.
There is a separate setup (“ucrt3”) created and maintained by Tomas Kaliberawith a new toolchain Rtools42 (GCC 10), with patched R-devel snapshot binarybuilds, patched CRAN package binary builds, patched Bioconductor packagebinary builds (only those needed by CRAN), and a compatible build of JAGS andTcl/Tk bundle. Automated R-devel binary builds and CRAN package checks have beenprovided sinceMarch 2021,with results linked from CRAN.More information is provided inHowto: UTF-8 as native encoding in R on Windows.To help package authors with testing and fixing their packages, “ucrt3”R-devel builds automatically apply patches created by Tomas Kalibera to somepackages at installation time (patches for over 100 CRAN packages andseveral Bioconductor packages have been created). Authors have beeninvited since March 2021 to adopt these patches, and there are features,described inHowto: UTF-8 as native encoding in R on Windows,allowing packages to do so while still supporting the current R releases.
Switching to UCRT
The “ucrt3” system is ready to be merged into the CRAN systems and R-develsource code. The process is planned to happen as follows and may take sometime - it is a more challenging change than previous toolchain upgrades.
CRAN systems being extended by Uwe Ligges are almost ready as well: binarypackages are already built and used for the purpose of CRAN package checks(results are already available on CRAN pages) and for checks via theWin-builder service.
On Monday December 13, CRAN will switch the incoming checks on Windows to what is now “ucrt3”. Atthe same time, R-devel source code will be patched with “ucrt3” patches.From that point on, it will assume 64-bit UCRT and no longer support MSVCRTnor 32-bit targets, and CRAN will start building R-devel snapshot binarybuilds with Rtools42. This switch should take from a few hours to a maximumof several days. During this short period, it might be difficult to buildR-devel from source, install binary packages in R-devel or submit packagesto CRAN.
The best course of action for package authors and users using R-devel onWindows will be to uninstall R-devel, uninstall old Rtools, delete the oldpackage libraries, and install the new versions from scratch. Those whobuild R-devel from source will have to run distclean.
After the switch, R-devel will be automatically installing patches for CRANand required Bioconductor packages at installation time, as “ucrt3” doesnow. This feature will be used temporarily to give package authors moretime to fix their packages. Eventually, patching a package at installationtime may be turned into a warning and the patches may be removed.
The switch is being coordinated with the Bioconductor team, who will eventuallyprovide full support for Bioconductor packages again after the switch ofR-devel and CRAN, but it is expected it might take a few days to geteverything synchronized.
Preparing for the switch
Authors of packages failing the UCRT checks and of packages with installationtime patches areinvited to already start adapting their packages. The check results havebeen available on the CRAN results page for each package since March (e.g.Matrix)and now the corresponding check flavors are r-devel-windows-x86_64-new-TK andr-devel-windows-x86_64-new-UL. The differences between the two are causedprimarily by different setups of the systems and package authors primarilyshould care about the latter run by Uwe Ligges, as this is the setup thatwill be used after the switch.
Package authors may now use this setup via theWin-builder service run by UweLigges to check their packages. The “ucrt3” system has also beeninstalled on R-hub about a month ago byGabor Csardi, where it can be used for building and checkingpackages. It is also available for use with github actions, seeGithub ucrt3 release andactions. This can be used togetherwith github actions provided by SimonUrbanek to check packages ondifferent platforms.
Before installing “ucrt3” locally, it is safest to uninstall any previousinstallation of Rtools and R-devel, and to delete the R-devel package libraries.The installers are available for“ucrt3” R-develand Rtools42.See Howto: UTF-8 as native encoding in R on Windows for detailedinstallation instructions and advice for package authors.
Most of the required package changes were due to downloading incompatiblepre-compiled libraries at installation time. Rtools42 includes librariesfor almost all CRAN packages, which can and should be used, instead. Usinglibraries from the toolchain ensures that they are built in a compatibleway, makes the source packages more transparent, removes download issues(which are perhaps rare for individual users but not so in CRAN operations)and makes bigger changes to the toolchain, like this one, much easier.
Outlook to further developments in the toolchain
Currently, the compiler toolchain and libraries are cross-compiled on Linux(using MXE). We could eventually support the fullinstaller build of R on Linux as most of R can already be cross-compiled,and it would be nice if we could also eventually cross-compile most or allof the R packages. This could make some operations faster, easier toautomate and easier to replicate for some uses. It is perhaps notsurprising that some build systems used in software primarily developed forUnix, which is the vast majority of software R packages use, run way fasteron Linux. Cross-compilation is used for instance by Julia and Octave (thelatter even using MXE). Even though it would have to be seen how big thebenefit would be there, cross-compilation could run even in WSL in Windows.
The distribution of the libraries for packages in a single chunk (or two,the “base” and “full” version) is subject to an ongoing debate, withdiffering opinions. Should distribution in smaller chunks (sets oflibraries, or even individual libraries) become necessary, it should beusing a package manager external to R packages and be integrated withthe toolchain bundle/Rtools, allowing automated re-build and changetogether with the compiler toolchain. MXE itself allows building.deb packages of the individual libraries, so that would be one option.
Recent changes
This is a result of ongoing work presented in earlier blog posts fromMay 2020,July 2020,March 2021.
The progress in “ucrt3” since March 2021includes:
Rtools42
The bundle includes Msys2 for builds tools (e.g.make) and the new compiler toolchain (for UCRT) and libraries. Theinstaller is almost the same as in Rtools4 and re-uses its code.
The differences are as follows. Rtools42 contains libraries for almost allCRAN packages, which allows to get rid of downloading pre-built libraries atpackage installation time. Rtools42 is one step easier to install: one doesnot have to put the compilers on the PATH. Rtools42 no longer has a specialimplementation of tar. Rtools42 only has a 64-bit compiler toolchain.
Github actions
“ucrt3” is also available on githubfor use with github actions. One may download the R builds and thetoolchain from github, using the provided actions, and check packageswith it.
For the purpose of github actions, there is also a new “base” distributionof the toolchain, which only includes the compiler toolchain and librariesneeded to build base R, but that is already enough to build many R packagesas well, and it is smaller than the “full” distribution.
Automatic construction of linking orders
Scripts are now available for package authors to automatically find/suggestwhich libraries to link and in what order to their packages. Withcontributions from Deepayan Sarkar. More inHowto: UTF-8 as native encoding in R on Windows.
Improved automation
The automated package checks and builds are run inside a docker container(on Windows and Linux). This allows to test that all installation ofexternal software is really automated, and hence done in a way that isrecorded. Also it shows that probably all CRAN R packages can really bechecked in a Windows docker container, so without some of the Windows GUIAPIs and without the GUI itself, which hasn’t at all been clear at thebeginning and this is a property worth preserving. The scripts used for“ucrt3” are availablehere.
Improved coverage
Even though it is a moving target, the check results are already comparablewith CRAN checks on other platforms, including the existing Windows checks.The toolchain has been upgraded to newer GCC, MinGW-w64 and a number oflibraries (geospatial an other) were upgraded as well. Inevitably, some ofthe package check issues were due to different setups between the checksystems waking up issues in packages that had not been discovered before.
Reduced size
Size of the toolchain has been reduced by about 1G by removing unnecessary executables,which were built by default by MXE, but not used by CRAN packages. That wasa significant improvement due to static linking, which makes the executables large.Also, the compressed size has been reduced by using a better compressiontool and re-ordering the files before compression to increase chances ofcommon parts being found by the compression algorithm.Now the “base” tarball of the compiler toolchain and libraries takes about100M compressed and 1000M uncompressed, the “full” tarball takes 360Mcompressed and 2.7G uncompressed, and the installer for Rtools42 takes about360M (the EXE file with internal compression) and 3G after installation.
