# WorkingWiki/ChangeLog

This page lists recent changes to the WorkingWiki software (formerly known as ProjectExtension) that might affect users.

I started this on July 21, 2009, so it won't have older changes. It will never be as comprehensive as the log messages on the svn repository, and rightly so. This page is only for noting changes that are directly of interest to WW users and administrators.

## 2011

• April 26: Changes for the unreleased MediaWiki 1.18alpha release. The Math extension isn't part of the core release any more, so we can't rely on its "Use MathML if available" preferences option. So now we have our own mathml option, and the administrator chooses whether to use ours or theirs.
• April 18: We've begun translation of WW's interface into Chinese. Russian may be in the works as well.
• April 11: Made the source code public by migrating the WorkingWiki subversion repository from lalashan.mcmaster.ca to sourceforge.net. The home page for the project there is https://sourceforge.net/projects/workingwiki/, there are links to the issue trackers and mailing list at the WorkingWiki page here, and the source code is available for download via
svn co https://workingwiki.svn.sourceforge.net/svnroot/workingwiki/trunk workingwiki
• April 11: Found a way to go back to short, friendly directory names in the .tar.gz package created by the export-project actions.
• March 30: A change to the default behavior of Special:GetProjectFile: if the display argument isn't given, serve the raw file for download, not a link in a special wiki page. It seems more convenient. If you want the page with a link (for instance, so you have access to the "make" action for the file), you can get there using display=link.
• February 28: Much improved support for editing and previewing sections of pages that include WW files.
• February 17: Better listings for background jobs, including the name of the target, the .make.log file, and the time of the listing.
• January 17: Support browsers that cache images, css files, etc. by allowing them to avoid reloading if they haven't changed.

## 2010

• December 27: A command line script, getprojectfile, allowing you to write scripts that pull files out of the wiki.
• November 19: Allow external projects (things that are stored in git or svn repositories, or in other wikis) to be prerequisite projects. ProjectEngine will pull them in from their repos and make the files available for your make jobs the same as with any prerequisite project. (It doesn't pull from other wikis currently though - so you need the other wiki to be connected to the same ProjectEngine cache and to sync up its own source files before you can use them.)
• November 19: New makefile $(RESOURCES)/makefile, which you can use when making a recursive call to make in a different project directory, to make it behave the same is when called by ProjectEngine, using the default makefiles in the right order. Don't use it with a project that doesn't use the default makefiles. • October 25: Major changes to WorkingWiki, which have been in progress for months (this new version was rolled out to lalashan wikis sometime later than October). It is now split into two separate components: ProjectEngine manages make jobs in working directories, and WorkingWiki takes care of storing and retrieving source files in wiki pages and displaying output files. This is because the WorkingWiki project provides three separate functions: a lightweight code repository (storage of source files and their histories), a lightweight integrated development environment (the graphical interface to the edit/build/view cycle), and a new thing, which is the ProjectEngine: receiving source files, running the computations, and returning the resulting output files. We should be able to swap out any of these three components for something else: using a wiki for development but storing the files in a git repository; using a single wiki with multiple ProjectEngines; using ProjectEngine on its own to support other clients such as a DokuWiki front end, or a Google-Docs type of service, or who knows what. Most of these things are in the future but these changes are to make them possible. • July 31: Customizable display rules and links for various filetypes. That is, the machinery that decides to make and display a .xhtml file in place of your .tex file, and attach links "[log,pdf]" to it, while other files display as themselves and just have a "[log]" link. Now this is controlled by variables that wiki admins can customize, to add special behavior for other file types. • June 21: Finally, a fix for updating project files when a source file is updated using the "Upload file" interface. This works by marking all the project's pages as containing a link to the uploaded file. As a consequence, all those pages will be listed at the bottom of the uploaded file's page, where it says what links to it. → This fix will take effect gradually, not all at once, as the pages are re-evaluated by the wiki's parser, so please don't be alarmed if it doesn't seem to be working at first. • June 15: A new setting on the ManageProject page: Use WorkingWiki's supplemental make rules. If you switch this off and click the "submit" button, the project will only use your makefiles, not the extra makefiles supplied by WorkingWiki for processing .tex files and images. This will also exclude the site makefiles' rules for R and other processes. • June 11: Split the internal makefile into two, so one can be read before the project's makefiles, and one after. This allows us to define variables early and give the project a chance to provide overriding definitions by putting them second, while defining pattern rules late, so that the project can provide alternative rules that override ours by putting them first. The makefile was called default-rules.makefile; now the two parts are called makefile-before and makefile-after. Likewise the old site/site-default-rules.makefile is now site/makefile-before and site/makefile-after. The old filenames are not used anymore. • May 7: All theobio wikis now have the background jobs feature enabled. The only visible difference is the "make in background" button on your project management page, unless you make use of the feature by creating project-file tags with make="background" or have background jobs running at the moment. • April 27 : More changes to make rules for .tex files. Now should be much easier to use pdflatex. Instead of the special makefile code we had to install in the project previously, now you just put pdflink="filename.pdflatex.pdf" into your source-file tag, and the [pdf] link will give you your paper via pdflatex. Added to the documentation here. • April 27: New link-related attributes in source-file and project-file tags: for example altlinks="log,dvi,pdf,foo", pdflink="filename.pdflatex.pdf", dvilink="filename.dvi", foolink="filename.foo". Also to customize the text of a display="link" link: <project-file filename="myfile" display="link" linktext="Click here to see my file"/>. the linktext attribute is added to the documentation here, and the other ones are in the new section The alternative links. • April 19 : Changes to default make rules for .tex files. Now the rule is to run latex as many times as needed (actually, run $(LATEX), which could be changed from latex to something else), then find out what it did. Previously, it was assumed to produce a .dvi file, and if the .dvi file wasn't there to be converted to .pdf it was an error. Now it can produce a .dvi, .pdf, or (in case a XeTeX user wanders in) .xdv files, and they'll be smartly made into .pdf. This is especially useful in the case of a file that runs fine under plain latex but contains a \pdfoutput=1 command or equivalent, which essentially transforms latex into pdflatex, causing it to output a .pdf file rather than a .dvi. With the new makefile, the Tips and Tricks entry for making WW use pdflatex or another variant is simpler than before. You only need to assign $(LATEX) and maybe $(LATEX_IMPLICIT_FIGURETYPE), and don't have to redefine the %.pdf : %.tex rule.
• April 6
: A background jobs feature is under development. It will probably become active on some lalashan working wikis soon.
• March 31: Added an inter-project dependency example. You can change the R code that generates an image file in one project, and the figure within a latex paper in a different project will update automatically when you reload the wiki page.
• March 10: Improvements to the timeout feature and related code. Wiki site administrators can now set $wwSecondsAllowedForMake = 0 to allow make jobs to run as long as they want to. Since this means you may end up wanting to kill a make job by hand, I also made a related bug fix: if you kill the ww-make process — this is the C program that executes make on behalf of WorkingWiki — it makes sure to kill its child processes before exiting, since this is what you'll want it to do. • February 24: Overhaul of importProjectFiles.php, the command-line tool, to match the features added to the ImportProjectFiles page of the wiki. In the process, the algorithm for suggesting pages changed a little on the wiki pages as well. The importProjectFiles.php script now allows importing from .tar.gz files and the other package formats supported by the wiki, and has a --defaults option for the optimistic. The hope is that a command like sudo -u apache php$(WIKIDIR)/maintenance/importProjectFiles.php --conf=$(WIKIDIR)/LocalSettings.php ~/exported-project.tar.gz --defaults  can be used to import a project exported from another wiki in a single step, and sudo -u apache php$(WIKIDIR)/maintenance/importProjectFiles.php --conf=$(WIKIDIR)/LocalSettings.php ~/working-directory --defaults  can be used to "check in" a local working directory after the project's been "checked out" and modified. • February 4: New file locking code. The code we had, using flock(), doesn't quite work on NFS-mounted filesystems. We now have an alternative, enabled by setting $wwUseFcntl in the php code, that makes it work on NFS, but it requires an extra package installed called php-dio.
• February 3: Overhauled import and export features. "Import Project Files" is now smart about the locations of project files that already exist. The Export operations bundle project information in the package they output, so that the page locations are there when you import it into a different wiki. The export package also includes a copy of the resources/ directory, allowing you to do the same make commands you can do on the wiki, using WorkingWiki's make rules and utility programs. To make use of the resources, begin your make command with make -f .workingwiki/makefile in place of just make. This revision also fixes two major bugs: one that stores the wrong page locations when you import a file, and one that causes ImportProjectFiles to skip some of the files in a package.
• January 19: Validation of project files is disabled for now. It needs improvement.
• January 7: Stop checking project files for dangerous content when they are displayed using the syntax highlighter (see September 24). I believe the syntax highlighter does enough escaping in its output that it won't let anything dangerous through. This change should get rid of a lot of irritating warnings about suspicious files that are actually harmless.
• January 7: Bug fix for {$...$} syntax that might have made it not work before. If you had trouble with it, try again.
• January 6: The site-specific makefile (site-default-rules.makefile) is now expected to be in resources/site/ instead of just in resources/. I recommend putting your other site-specific files in that subdirectory too, to make maintaining and upgrading easier. A makefile variable is provided, $(SITE_RESOURCES), containing the name of this directory, so you can use that instead of $(RESOURCES) to point to those files.
• January 6: The machinery that inserts suggested page locations when you are adding a source file on the ManageProject page was broken. Now it's fixed and improved, and works for archived project files as well as source files. The new version uses the same code as the suggestions machinery on ImportProjectFiles, so the two ought to be consistent with each other. More improvements to this code are on the way, including knowing where source and project files are already located, if they have locations assigned.

## 2009

• December 30: Bug fix in makefile rules for .tex files. From now on you can count on "%.tex" being a dependency of "%.latexml.xhtml" and "%.pdf", so you can make more complex rules that produce .tex files on demand, and so forth.
• December 18: Another change to directory locking logic (see November 24). These shouldn't affect any WW users. Now the flow is this, for those who are interested: first the parser goes through the wiki page, and WW registers all the project files involved; then at the end of parsing WW locks all the affected projects and their dependency projects, and does all the syncing into working directories, make operations, and reading file contents out of the working directories, then unlocks everything. Any project files that are going to be archived are copied to a temporary directory just before unlocking. Then after all the unlocking and parsing, it inserts the archived files into their pages and saves them, because the save step involves another round of parsing — which requires locking and everything, but WW knows it's in the middle of archiving and doesn't redo the making and archiving because that would be ridiculous. When a set of projects is locked, they are locked in alphabetical order and unlocked in reverse order, to prevent deadlock between multiple WW processes. That's why we have to have a second phase for the parsing-and-saving, because we can't predict which projects will be involved, and so locking them as they come into play would violate the alphabetical order, leading to deadlocks, in which one person's page request is waiting for a lock held by another person's page request which is waiting for a lock held by the first one... this condition would freeze up the wiki and require someone to restart the web server manually.
• December 17: New setting $wwMaxInsertFileSize, to limit the size of a source file or archived project file that is inserted into a textual wiki page by the program. You can still put a longer one in when editing by hand. • December 17: Move the clear and sync all actions to the top of the directory listing page, and add export actions. • December 11: Bug fix so that uploading a file into an existing tag doesn't erase the attributes you've added to that tag, like make="yes" or display="source". • December 9: The makefile now includes a variable $(LATEX_IMPLICIT_FIGURETYPE). This is normally set to "eps", because when latex sees \includegraphics{fig1} it looks for a file called fig1.eps, but when for instance you set LATEX=pdflatex you can also set LATEX_IMPLICIT_FIGURETYPE=pdf because fig1.pdf is the default. For your reference, here is what to put in your project's makefile to make your tex files compile using pdflatex instead of latex:
LATEX=pdflatex
LATEX_IMPLICIT_FIGURETYPE=pdf
%.pdf : %.tex
$(run-latex)  • November 24 : Change to directory locking protocol (see October 4): now a project's working directory, along with those of its prerequisite projects, is locked the first time it is used, and stays locked until all WW operations on the current wiki page are done. This reduces potential redundant syncing operations, and helps guarantee that all project files on the page will be consistent with each other (rather than allowing some other user's update to source files after some of the page's operations and before others). It doesn't guarantee consistency of project files that are images though, because those are requested separately by the browser sometime after the rest of the page. • November 24: Added support for inter-project dependencies. The ManageProject page now has a place to record projects that the current project depends on, and those projects will be synced and locked when the current one is, so you are guaranteed their working directories will be up to date. (This only works among projects on the same wiki.) • November 24: Rearrangement on ManageProject page. Moved some things to higher on the page, and added an "Import Files" link. • November 23: Lines of text now wrap in source code listings instead of running off the right edge of the page. • November 18: Source code listings of WW files now include the filename as part of the blue dashed border (on regular wiki pages, not on the GetProjectFile page where it would be redundant). • November 11: Internal changes in how .wikitext files are parsed. This should allow more things to be parsed correctly. It may cause some existing wikitext to be handled differently from before, and if any changes are for the worse please make a bug report. • November 6: Now using nice to lower WW make jobs' priority. This should allow intensive make jobs to run with less disruption to other things that are happening on the same server. The nice level is controlled by a variable $wwNiceValueForMake, whose value is 0 by default, but we are using a value of 10 on our mcmaster wikis, which makes them more retiring in comparison to other processes.
• November 4: Now using latexmlmath command to translate short math snippets (the ones marked by $$...$$ and {$...$}) to HTML, instead of using the regular latexml command, because it runs faster. If this produces any disruption in which packages are available by default or anything else, please report the bug.
• November 3: More makefile reordering such that site-default-rules.makefile (the one optionally added by the administrators at your site) is read at the end of default-rules.makefile (the one that comes with WW) rather than at the beginning. Also added some useful variables: $(LATEX),$(DVIPDF), so you can replace the default executable names with other ones in the standard rules.
• November 3: Bug fix allowing completely empty source files.
• October 26: Project makefiles are now read after the default makefiles, not before. This allows you to put pattern rules in your project makefile to overrule the ones in the standard makefiles. Also, the rule for making .pdf files from .tex files is changed in the default makefile so that it's a single rule, instead of one that makes a .dvi and a second one that converts it to .pdf, so that project editors can put in a replacement rule using pdflatex or some other process.
• October 23: Changed rules about symbolic links. Your makefile can create symbolic links from the project directory to some other project directory, but not to arbitrary files on the server. You aren't allowed to make source files be symbolic links — they'll be made back into plain files.
• October 19: New remove-and-then-remake action on ManageProject page and directory listing page. This is implemented by removing the file and then redirecting you to a GetProjectFile page that does the make, so reloading that page redoes the make but not the remove — be warned.
• October 19: Added a [make] link to make your file from the GetProjectFile page, when the page isn't making the file already and it isn't a .make.log file. This is useful in conjunction with the links in makefiles (see September 30). Also, the "raw" link has been renamed to "download".
• October 14: I may have fixed the bug that puts extra line breaks in the Edit form controls! That's been bothering me for a long time.
• October 4
: Now using the flock() system call to lock the project's working directory when there's a make operation in progress, and when reading from the project files, to prevent collisions when two wiki users are trying to access the project's files at the same time. This has been part of the plan for a very long time, but there was a bug in the way which is now fixed.
• September 30
: Target and dependency names in makefiles are now links to the project files by the same name. It works as long as your makefile isn't too creatively written. You can see it working for example in the makefile on the WorkingWiki/Script Example page.
• September 30: Started outputting warnings where "remake" is used saying to change it to "make" for future compatibility (see August 19).
• September 29: A significant new feature called archived project files. This allows you to mark a project file to be archived on a particular page (either a textual wiki page or an image page), and whenever the file contents change in the working directory, the file will be saved onto the wiki page, giving you a record of the current and old contents of the file. Also, you can export the project to your home directory, edit the source and run make jobs yourself, and then re-upload the updated source and project files to the wiki together. For more information see the archived project files section of the documentation.
• September 24
: An important security feature: verifying project files' content for anything that could endanger the wiki user's system. Among other things, blacklisted types of files will be blocked rather than displayed from the working directory (.exe files, for instance), and files containing JavaScript code that could subvert a browser's security will not be displayed. For more information see the Security page. (Note: we have this feature enabled as an advisory only, in the sense that it will warn if a file appears suspicious but will go ahead and display it. This is for a breaking-in period, to make sure that false positives don't get in the way of work.)
• September 24: It is now possible to use different chroot jails (or different working directory locations, if you don't use chroot), for different wikis using the same installation of WorkingWiki. It's generally possible to customize the WW settings from the LocalSettings.php or equivalent file now, which includes the chroot directory location as one case.
• September 9: It's now possible to run WorkingWiki without chroot. This is a convenient option for closed wikis whose users are trusted not to mess up the server system. Also, there are probably better ways to secure your server than using chroot — I'm thinking of rule-based access control — so we may end up not using it.
• September 3: We now serve MathML pages to fewer users. Until now we were sending MathML content to any web browser that is capable of accepting it, whether the user wants it or not. This makes it look good when people are set up to view MathML. Unfortunately, if we do it that way, a lot of people who are using Firefox but haven't installed the fonts for displaying MathML will get wiki pages with annoying warnings and illiterate-looking equations. So we've changed to a more logical system where MathML only goes to users who have logged in to the wiki and selected the MathML option in their user preferences. Everyone else gets regular HTML pages with images made from the latex code. We'll have a screenshot here on the projects wiki (soon) (done Sept. 4 –LW.)to show them what they could be seeing. Users who are logged in and using a MathML-capable browser, but not using MathML, see a notice advising them how to enable it.
• September 2: Changing the name of the extension from ProjectExtension to WorkingWiki. First is to change the documentation, then I will change the names in the actual code. (Completed Sept. 7 –LW.) The change includes CSS class names: all names beginning with "pe" now begin with "ww" instead. Any user CSS relevant to these classes is now broken and needs to be updated. Not that I think anyone has made any user CSS.
• August 27: No more asking whether to add or remove project files when you edit a page. Instead we now have warnings that arise afterward, when a new file is encountered or an old one is missing. You are given the option to update the project at that time.
• August 26: Inline LaTeX snippets (i.e. $$and <latex> items) aren't processed inside <source-file> elements any more. This means you can use$$ in your source-file text without having it change into something unwanted. Inline LaTeX is interpreted in .wikitext files, when they are passed through the parser. Also, {$...$} is re-enabled, as a synonym for <latex>$...$</latex>.
• August 24: The "projects" box on the sidebar now includes a "suggested" project name if there is no actual project in use on the current page. Also, if a project doesn't exist (yet) the project link will be red.
• August 19
: The "remake" attribute for <project-file> tags (see July 31) is renamed to "make". So is the "remake" argument to Special:GetProjectFile. Both still accept "remake", but eventually they won't any more, so please change your pages. Sorry for the inconvenience.
• August 19: When a project file that's an image is missing, it appears in Firefox now as its filename in parentheses, instead of just nothing there. This is accomplished by attaching an alt attribute to the image tag, and making sure missing images are served properly (i.e. a 404 Missing page, not a wiki page with error messages on it).
• August 18: fixed a bug that was causing it to fail to update pages that include project files when the source files have changed.
• August 18: Project makefiles can now define LATEX, BIBTEX, DVIPDF and a few other names of executables that are used in the default rules to process latex documents.
• August 14: improvements to the "remove" action: now it doesn't ask whether you want to remove the file from the project if it's not in the project description, and doesn't ask about removing it from the wiki if it isn't stored on the wiki. If neither of these options applies, it just removes it from the working directory in one step with no questions.
• August 11: added "Export working directory" as an alternative to "Export source files", which is the "Export project" feature we already had. "Export working directory" gives you a .tar.gz containing whatever's currently in the working directory, including target files.
• August 10: changed rules for updating working directory: now if a source file in the directory is different from what the wiki thinks it should be, it'll be replaced before doing any makes. The only time it's left alone is if its timestamp is the same as the page's or it has the same bytes in it as the version on the page. Before it would be left alone if it's as new as or newer than the wiki page it comes from. This should help with some of the confusing behaviors involving previewing, undoing edits, and viewing old revisions of pages.
• August 6: removed a few extraneous targets like all from default-rules.makefile. They shouldn't be defined in projects that don't define them explicitly.
• July 29: "Sync" and "remove" now stay on whichever page you called them from (ManageProject or GetProjectFile). Before they took you to ManageProject regardless. Also, the "make" button is suppressed for .make.log files.
• July 27: There was an out-of-date version of R in lalashan's PE system. Apparently we upgraded at some point from an R installation that had the main executable in /usr/bin/R to one that places it in /usr/local/bin/R; the old copy was still in chroot-trap/usr/bin/R and getting run instead of chroot-trap/usr/local/bin/R. I removed that and fixed up the dynamic library dependencies, and it seems to be running all right now.
• July 23: Noticing some glitches when editing latex papers, at least on theobio/projects: the wiki page doesn't keep up to date with the changes. Will track down this bug very soon. It must be a side effect of something else I changed.
• July 22: checked the tests for edit permission and added new ones where needed. This allows us to give read-only access to unregistered visitors to the wiki, and along with being allowed to view but not edit pages, they are also allowed to view project information (the ManageProject page, working directory contents, project files), but not do any actions like make, sync, import. The current implementation is that if the visitor doesn't have permission to edit pages, they also aren't allowed to do anything that modifies projects, project files, or working directories. We could invent a new kind of permission — "edit-project" rather than "edit" or such — but it doesn't seem necessary.
• July 22: bug fix: if you start with a project that's implicitly defined by the source-files on one page (it doesn't have a project description), and then you allow PE to add one or more files to it when you edit and save a page — this creates an explicit project description — it would only put the new files in the project description. Now it includes all the files that were implicitly in the project as well.
• July 20: I've updated to the current version of the SyntaxHighlight extension and am using it differently, so that source and project files now appear in dashed boxes, similar to the boxes created by the <pre> tag. I wanted to do that from the beginning. I hope it doesn't cause much formatting trouble. Feedback is welcome.
→ Related to SyntaxHighlight: apparently there's a way to get it to wrap long lines, when it's listing files with line numbers; but it doesn't work on our wikis because it breaks the XHTML parsing. I'm going to have to pursue it as a bug report to one of the authors, or see if the problem goes away when I've done some rewriting on my end to allow us to enable the HTML tidying feature in the wiki...
• July 20: fairly comprehensive overhaul of the documentation on theobio/projects.

WuLi