VSoft Technologies BlogsVSoft Technologies Blogs - posts about our products and software development.https://www.finalbuilder.com/resources/blogsDPM Package Manager - Progress updatehttps://www.finalbuilder.com/resources/blogs/postid/842/dpm-package-manager-progress-updateDelphi,DPMWed, 24 Feb 2021 10:46:15 GMT <p>In December 2019, I <a href="/resources/blogs/introducing-dpm-a-package-manager-for-delphi" target="_blank">blogged</a> about a package manager for Delphi that I am working on. This post is a progress update that shows where it's at and what's left to do to get to v1.</p> <h3>DPM Recap</h3> <p> For those not familiar with what I am trying to achieve here, I highly recommend reading my original <a href="/resources/blogs/delphi-package-manager-rfc" target="_blank"> Delphi Package Manager RFC</a> post. In that post I detailed my ideas, and some of the challenges that Delphi presents when compared to other development environments. </p> <p> In December 2019, the bare bones of DPM were there. We had a command line tool and that was it. We were able to create packages, install packages (and their dependencies) and restore them (restore ensures all referenced packages are present). Oh and we could list the available packages in our package feed (a folder). </p> <h3>IDE Integration</h3> <p> In the last 13 months there were around 175 commits to the DPM repository. In that time I have added an IDE plugin (that works in Delphi XE2 to 10.4). This involved the creation of several custom controls (I wasn't able to bend any existing ones to work how I wanted it to). </p> <p> In addition to the work in the project repository, I also published <a href="https://github.com/vsoftTechnologies/" target="_blank">several useful libraries</a> that I needed for this project. DPM is now bootstrapped, to build DPM you need DPM, as it requires several libraries that are referenced as dpm packages. </p> <p> In Nov 2020 I published the first alpha release that included an installer (code signed by VSoft Technologies) for installing both the command line tool and the IDE plugin (single installer, you can choose which IDE versions to install for). The installer allows you to install for the current user, or for all users (requires elevation to install). </p> <img src="/blogimages/vincent/dpm-progress/installer.png"> <p></p> <p> I also did a zoom presentation about DPM to the Melbourne chapter of the Australian Delphi Users Group - a recording of that (long) presentation can be found <a href="https://www.youtube.com/watch?v=TjVAMLfhgLo" target="_blank">here</a>. </p> <p> Adding IDE support for DPM was a massive undertaking. I had very little experience in developing Delphi IDE plugins (using the tools api) - and there were lots of subtle changes between delphi versions, getting things working correctly in 12 versions of Delphi was not easy. In particular, with the later versions of Delphi IDE that use VCL themes, getting things to look right (ie like a native part of the IDE) was a challenge. </p> <img src="/blogimages/vincent/dpm-progress/dpm-ide.png"> <p></p> <p>The above image shows the installed packages for one of the projects in the project group, you get to this view by right clicking on the project node, or the DPM Packages node in the Project tree. </p> <p>Note the view only shows the directly installed packages, not the transient dependencies - those you can see in the project tree under the DPM Packages node.</p> <p>Before you can use DPM in the IDE, you need to configure a package source (a folder where your package files will live)</p> <p>This can be done fron the command line</p> <pre>dpm sources add -name=local -source=path to the folder you created</pre> <p></p> <p>Or from the IDE Settings</p> <img src="/blogimages/vincent/dpm-progress/dpm-settings.png"> <p></p> <h3>Compile during install</h3> <p> The most recent updates added support for compiling packages during first install. Packages need to declare how to build in their dspec file, and dpm will use that and call msbuild to compile the packages if needed. DPM also records a bill of materials file (package.bom) in the package cache so that it can tell whether the package needs to be recompiled or not. </p> <p>On first install, packages that are being compiled during the install process will take a little longer, but on subsequent installs or restores, the process is almost instant (a few ms). </p> <p> Prior to adding this feature, building dpm on our Continua CI build agents took 13 minutes, much of which was taken up with compiling the dpm packages that it references (in particular, earlier versions of Delphi were very slow with spring4d). Since updating dpm on our agents with the new version, the entire build process for DPM (console app and 12 versions of the IDE plugin and the installer) takes less than 2 minutes. </p> <img src="/blogimages/vincent/dpm-progress/dpm-build-times.png"> <p></p> <h3>Missing features</h3> <h4>Project group support</h4> <p> When installing packages, the dependency resolution code does not know about other projects in the project group, or what packages and versions they reference. This will be a problem for packages that include design time components that need to be loaded - the IDE can only load 1 version of a design time package. This is what I am currently working on. </p> <h4>Design time packages</h4> <p>DPM does not currently install design time packages into the IDE. This is dependent on project group support, so it's next on the list after project group support.</p> <h4>Package Updates</h4> <p>The ability to detect when package updates are available and make it easy to install those updates. There's an Updates tab in the IDE but it's non functional at this time.</p> <h4>Package Repository</h4> <p>In it's current state, DPM only supports folder based package feeds. This works fine, but it does have some limitations</p> <ul> <li>Limted search abilities - limted to searching on the package filenames.</li> <li>You have to download packages to a folder.</li> <li>Package Authors have to host the package files somewhere (mine are under releases on their github projects).</li> </ul> <p>I have made a start on the Package Repository, but not a lot of progress since I'm focusing on the client site right now.</p> <h3>Q & A</h3> <h4>Is it usable?</h4> <p>In it's current state, it's only usable for non visual libraries. As I mentioned, the DPM projects all use DPM themselves, and we have DPM actions in FinalBuilder for running the Pack and Restore commands. </p> <p>If you use any of my open source libraries like DUnitX, Delphi Mocks etc, I have created packages for all of those libraries, and also created mirror projects (just for hosting the package files) for some other popular libraries like Spring4D. </p> <p>I would encourage library authors in particular to take a look and provide feedback.</p> <h4>Where can we find it?</h4> <p>DPM is an open source project on <a href="https://github.com/DelphiPackageManager/DPM" target="_blank">GitHub</a>, the installer can be found under <a href="https://github.com/DelphiPackageManager/DPM/releases" target="_blank">Releases</a> (under each release, there is an Assets dropdown section).</p> <img src="/blogimages/vincent/dpm-progress/installer-location.png"> <p></p> <h4>What versions of Delphi does it support?</h4> <p>Delphi XE2 to 10.4.2 - note that we compile with the latest updates installed for each compiler version. </p> <h4>Why is it taking so long?</h4> <p>Yes, someone asked that recently! This is a side project, free and open source. My primary focus is on running my business and working on our products (that keeps the lights on).</p> <h4>Can we sponsor the project?</h4> <p>Not right now, however it's something I'll look at in the future. </p> <h4>Can we help?</h4> <p>Absolutely. Fork the project on GitHub and clone it to your dev machine and spend some time getting to know the source code. Before making any pull requests, create an issue on github to discuss your ideas and make sure we on the same wavelength!</p>842Advice for Delphi library authorshttps://www.finalbuilder.com/resources/blogs/postid/841/advice-for-delphi-library-authorsDelphi,DPMWed, 24 Feb 2021 00:57:00 GMT <p>We use many third-party Delphi libraries to build FinalBuilder and Automise, and that brings plenty of issues when upgrading compiler versions. I've been using Delphi since 1995, both as a developer and as a component vendor, I have learned a thing or two about creating libraries that I would like to share. These are all ideas that make life easier for users, and make it easy to migrate from one version of Delphi to another.</p> <p>There's no hard and fast rules on how Delphi Libraries are <i>supposed to be</i> structured, these are just my preferences and things I have learned over the years. Hopefully this will help new and existing library authors.</p> <h3>Folder Structure</h3> <p>Keep the Source and the Packages in separate folders, this makes it easier to find the correct packages to compile, e.g : <pre>\Source \Packages \Demos</pre> </p> <p>Under Packages, create a folder for each compiler version your library supports, e.g: <pre>\Packages\Rad Studio XE8 \Packages\Rad Studio 10.0 \Packages\Rad Studio 10.1</pre> </p> <h3>Package Names</h3> <p>Please, <b>do not</b> put the Delphi version in the package project names.</p> <h4>Bad!!!</h4> <pre>MyProjectRun_D10_4.dproj MyProjectDesign270.dproj</pre> <h4>Good</h4> <pre>MyProjectRun.dproj MyProjectR.dproj MyProjectDesign.dproj MyProjectD.dproj</pre> <p></p> <p>Why not put the compiler version in the package project name you might ask? Well the answer is that it makes upgrading compiler versions a major pain for users who link their projects with Runtime Packages (yes, that includes us).</p> <p>The reason is that when you compile a package, it creates a packagename.dcp file and that is what your project references. So, if your package name is MyPackageRun_D10_4 then that is what will be added to projects that use it.</p> <pre class="brush:delphi; toolbar:false;"> package MyOwnPackage; //... requires rtl, vcl, MyPackageRun_D10_4, AnotherPackage_Sydney, YetAnotherPackage_D104, // ... </pre> <p></p> <p>When Delphi 10.5 comes out, guess what the user has to do to upgrade their projects.... Yep, replace that all those package references with 10.5 versions (and the multitude of suffixes). Multiply that by a number of projects and a number of libraries (each with potentially multiple runtime packages) and you can see why this might be a pain.</p> <p>Now you might say, but we don't want 15 versions of MyPackageRun.bpl laying about on users machines, and you would be right. The solution to this is a feature that has been around since Delphi 6 (2001) - <a href="http://docwiki.embarcadero.com/RADStudio/Sydney/en/Compiler_directives_for_libraries_or_shared_objects_(Delphi)" target="_blank">LIBSUFFIX</a>.</p> <img src="/blogimages/vincent/advice-for-d/libsuffix.png" alt="LIBSUFFIX"> <p></p> <p>Setting LIBSUFFIX (on the Description section of project settings) will append the specified suffix to the BPL file name. So a suffix of _D10_4 will result in a package :</p> <pre>MyPackageRun_D10_4.bpl</pre> <p></p> <p>however, the DCP file will still be generated as :</p> <pre>MyPackageRun.dcp</pre> <p></p> <p>Remember it's the dcp file that our projects reference (for linking) - so by keeping the dcp file the same for all delphi versions, upgrading to a new compiler version just got a whole lot easier!</p> <p>So when Delphi 10.5 comes out in the future, all I need to do is install the packages, no changes to my projects.</p> <p><b>Update</b> : Someone pointed out that Delphi 10.4.1 support LIBSUFFIX $(Auto) - this will use the Delphi defined PackageVersion - which for 10.4 is 270. This is a nice addition as it makes upgrading the package projects simpler. Of course if you don't like the PackageVersion suffix and use a custom one, then this is not for you.</p> <h3>Use Explicit rebuild, not Rebuild as needed</h3> <p>Have you ever encountered the error <pre>E2466 Never-build package 'XXX' requires always-build package 'YYY'</pre> What this means is, a package, set to Expicit rebuild, references another package, set to 'Rebuild as needed', and it's a pain in the proverbial. Rebuild as needed is also referred to as Implicit Build - in dpk's you will see it as <pre>{$IMPLICITBUILD ON}</pre> If that "Rebuild as needed" package is not part of your project group, guess what, you get to waste time closing and opening projects trying to get it to compile. </p> <p>I'm sure someone will correct me on this, but I cannot see a good reason to have "Rebuild as needed" set. I suspect this is a hangover from before the Delphi IDE allowed you to specify <a href="http://docwiki.embarcadero.com/RADStudio/Sydney/en/Project_Dependencies" target="_blank">Project Dependencies</a> and it slows down builds. </p> <h3>Use Search Paths for includes</h3> <p>I often see includes with either hard coded paths, or relative paths like this :</p> <pre class="brush:delphi; toolbar:false;"> {$I '..\..\MyDefines.inc'} </pre> <p>That's great, if the installer delivers the files in the right place - but they often don't - I hit this issue today, where the package just would not compile. I eventually figured out that the relative path was wrong. </p> <p>There's a simple fix for this, and that is to remove the path in the $I statement, and use the Project Search Paths feature instead. </p> <img src="/blogimages/vincent/advice-for-d/searchpath.png" alt="Search Paths"> <p></p> <p>I have also seen libraries where there are mulitple copies of the include file and they are slightly different!</p> <h3>Mark packages as Runtime only or Designtime only</h3> <p>Some libraries have their packages marked as "Runtime and Designtime" (the default) - the impact of this is only minor, but it's a pet peeve of mine. The Delphi IDE (in recent versions at least) provides a nice indication of whether packages are runtime or designtime in the project tree, and for designtime packages, whether they are installed. </p> <p>This makes it simple for me to determine which ones need to be installed or not.</p> <p>Not Installed</p> <img src="/blogimages/vincent/advice-for-d/not-installed.png" alt="Not Installed"> <p>Installed</p> <img src="/blogimages/vincent/advice-for-d/installed.png" alt="Installed"> <p></p> <h3>Summing up</h3> <p>One of the major reasons people do not upgrade Delphi versions is because it's too hard to deal with the third party libraries and all the changes required just to get to the point of compiling. That eventually results in a lack of Delphi sales which results in a lack of investment in Delphi which feeds back into.... well you get the idea ;)</p> <p>Making third party libraries easier to work with in Delphi has been a bit of a crusade for me, I've been <a href="/resources/blogs/delphi-package-manager-rfc">working on this for a while now</a>, and I'm getting closer to a solution - <a href="https://github.com/DelphiPackageManager/DPM">DPM - A package manager for Delphi</a> - if you are a library author, I encourage you to take a look. For examples on how to create a package spec (dspec) take a look at our open source projects <a href="https://github.com/vsoftTechnologies/">https://github.com/vsoftTechnologies/</a> </p> 841Introducing DPM - a Package Manager for Delphihttps://www.finalbuilder.com/resources/blogs/postid/837/introducing-dpm-a-package-manager-for-delphiDelphi,DPM,Open SourceThu, 12 Dec 2019 09:41:00 GMT<p>Back in Feb 2019, I <a href="/resources/blogs/delphi-package-manager-rfc" target="_blank">blogged</a> about the need for a Package Manager for Delphi. The blog post garnered lots of mostly useful feedback and encouragement, but until recently I could never find a solid block of time to work on it. Over the last few weeks I've been working hard to get it to an mvp stage.</p> <p>DPM is an <b>open source</b> package/library manager for Delphi XE2 or later. It is heavily influenced by Nuget, so the cli, docs etc will seem very familiar to nuget users. Delphi’s development environment is quite different from .net, and has different challenges to overcome, so whilst I drew heavily on nuget, DPM is not identical to nuget. I also took a close look at many other package managers for other development eco systems.</p> <h2>What is a Package Manager</h2> <p>A package manager provides a standard for developers to share and consume code. Authors create packages that other developers can consume. The package manager provides a simple way to automate the installation, upgrading or removal of packages. This streamlines the development process, allowing developers to get up and running on a project quickly, without needing to understand the (usually adhoc) way the project or organization has structured their third party libraries. This also translates into simpler build/CI processes, with less ‘compiles on my machine’ style issues.</p> <h2>Who and Why</h2> <p>DPM’s initial developer is Vincent Parrett (author of DUnitX, FinalBuilder, Continua CI etc). Why is discussed in <a href="http://www.finalbuilder.com/resources/blogs/delphi-package-manager-rfc">this blog post</a>.</p> <h2>DPM Status</h2> <p>DPM is still in development, so not all functionality is ready yet. At this time, it's at the stage where we I would encourage library authors to take a look and play with it and provide feedback (and perhaps get involved in the development). It's very much at a minimum viable product stage. Potential users are of course welcome to look at it and provide feedback, it's just that, well, there are no packages for it yet (there's some test packages in the repo, and I'll be creating ones for my open source libraries). .</p> <h3>What works</h3> <ul> <li>Creating packages</li> <li>Pushing packages to a package source.</li> <li>Installing packages, including dependencies</li> <li>Restoring packages, including dependencies.</li> </ul> <h3>How do I use it</h3> <p>The documentation is at <a href="http://docs.delphipm.org">http://docs.delphipm.org</a></p> <p>See the <a href="http://docs.delphipm.org/get-started/getting-started.html">getting started guide</a>.</p> <p>The command line documentation can be found <a href="http://docs.delphipm.org/commands.html">here</a>.</p> <p>The Source is on GitHub <a href="https://github.com/DelphiPackageManager/DPM">https://github.com/DelphiPackageManager/DPM</a></p> <h3>Is DPM integrated into the Delphi IDE</h3> <p>Not yet but it is planned. If you are a wiz with the open tools api and want to contribute then let us know.</p> <h3>Is there a central package source</h3> <p>Not yet but it is planned. At the moment, only local folder based <a href="http://docs.delphipm.org/concepts/sources.html">sources</a> are supported. The client code architecture has a provision for http based sources in the future, however right now we are focused on nailing down the package format, dependency resolution, installation, updating packages etc.</p> <h3>Is my old version of delphi supported</h3> <p>Maybe, <a href="http://docs.delphipm.org/compiler-versions.html">see here</a> for supported compiler versions. All target <a href="http://docs.delphipm.org/platforms.html">platforms</a> for supported compiler versions are supported.</p> <h3>What about C++ Builder or FPC</h3> <p><a href="http://docs.delphipm.org/compiler-versions.html">see here</a></p> <h3>Does it support design time components</h3> <p>Not yet, but that is being worked on.</p> <h3>How does it work</h3> <p>See <a href="http://docs.delphipm.org/concepts/how-it-works.html">this page</a></p> 837