drive v0.3.9 Release Notes

Release Date: 2017-01-07 // about 7 years ago
  • ๐Ÿš€ This is the first release for 2017, following the last release ~4 months ago.

    ๐Ÿš€ Lots of bug fixes and updates to this release:

    ๐Ÿ›  fixed a progress bar overflow: This bug was reported in duplicate issues:

    ๐Ÿ›  fixed up spurious pagination that resulted from asynchronous pagination before children were produced.
    This issue was reported by @bjohas in #724.
    ๐Ÿ›  It was fixed by PR #768.

    unshare no matches found bug when no --type was specified.
    This issue was reported by @bjohas in issue #747.
    The problem was that if a user made a request such as:

    $ drive unshare -emails [email protected] -id some\_id
    

    0๏ธโƒฃ without having specified the --type, no default accountType was set, hence the
    ๐Ÿ‘‰ user would get back an non-intuitive/cryptic error

    no matches found!
    

    ๐Ÿ›  The change in PR #771 fixed it; it adds User
    0๏ธโƒฃ as the defaultType.

    ๐Ÿ›  fixed a regression in which emails were no longer being set. This issue was noticed in #762. It was fixed by PR #763.

    ๐Ÿ›  fixed a bug in which local and remote dirTypes differed. There was a long standing TODO in code
    ๐Ÿ›  to fix this issue and it was high time that the chickens had come home to roost. This issue was fixed
    by PR #770.
    The result of the fix is that if the types differ, we now get an informative error

    $ drive new --mime-key docs share\_test $ mkdir -p share\_test $ drive pull share\_test; echo $?Resolving... /share\_test: local is a directory while remote is a file 8
    

    ๐Ÿ›  fixed up a README inconsistency in the documentation of the various options. This issue was reported by
    @jean-christophe-manciot in issue #772. The issue involved
    ๐Ÿ“ฆ contributions from @sselph and @shaggytwodope. The cause of confusion is that Go's flag package
    accepts both -flag and --flag indiscriminately. For folks used to *NIX CLI options get confused
    because they are accustomed to -s and --short instead where:
    +-s is the short option
    +--short is the long option
    ๐Ÿ›  The issue was fixed by @shaggytwodope with PR #779.

    โž• added the ability to structure .driverc files. This feature was requested by @jean-christophe-manciot in
    issue #778. The need for this feature is that using a single global
    ๐Ÿ”ง configuration for every option is very limiting, because one will want --depth=1 for list and --depth=-1
    ๐Ÿ’… for push, of which the old style of a single configuration style couldn't accomodate.
    The solution was implemented by PR #813 and
    ๐Ÿ“ฆ package https://github.com/odeke-em/namespace.
    ๐Ÿ’… The new style .driverc file allows for structure e.g

    id=true [global] depth=10 [pull/list] depth=2 [push] verbose=true [list] long=true
    

    where a section is a command's verb that can be run on the commandline e.g push, pull, list, delete, trash etc, as well as special global which indicates the fallback/general-purpose section.
    The rules of resolution are:
    a) If an option is specified in a section that isn't [global], it has precedence over the
    0๏ธโƒฃ [global] or default options e.g

    [push] no-clobber=true [global] no-clobber=false
    

    no-clobber=true wins
    b) for options in the same section, the later the option the higher the overriding precedence e.g

    [list] long=false depth=-1 long=true
    

    In this case long=true was declared later so long=true wins

    c) multiple sections can be specified in a section, but separated by "/":

    [push/pull/list] depth=-1 no-clobber=true
    

    will make depth=-1, no-clobber=true for push, pull and list.

    ๐Ÿ›  fixed an off by one traversal depth bug for md5sum and stat. This bug was reported by
    @jean-christophe-manciot in #783. The problem was that at
    the entry of the recursive function, we'd asap check if depth=0 and exist. However, if we are traversing
    a directory/folder, in order to stop recursion on every step we'll decrement the depth by 1. By the time we
    hit a non-directory after starting from a directory, we'd not stat/md5sum it. The fix was to first perform
    ๐Ÿ›  the desired operations and then only check that condition at the end. It was fixed by PR
    #832.

    ๐Ÿ“š Updated the README documentation to describe how to use --notify for file sharing. This issue was reported by @bjohas in issue #791. It was also fixed by @bjohas
    with PR #791.

    ๐Ÿ›  fixed an illogical return status code on stat failure. This bug was reported by @aueuaio in issue #801. The problem was that the stat error was getting shadowed and also it was being logged to stdout instead of being recomposed and propagated on exit. It was fixed by PR #804.

    โž• added a new mode to fix clashes with option --fix-mode. This feature was designed and implemented by @iblis17 in PR #810. The feature is a new policy that allows you:

    • trash clashing files.
      0๏ธโƒฃ The default behavior is to rename them.
      We can now do

      $ drive clashes --fix --fix-mode trash

    In that PR, also @jpambrun gave an alternate view and caution on user expectations.
    This is pretty cool, it was @iblis17's first Go code and they knocked it out of the park!!
    Congrats @iblis17, we are lucky to have you in the drive community and also in the Go community!

    ๐Ÿ›  fixed spurious rename error for non-existent-to-begin-with-file. This error was reported in issue #826. The problem and use case was that I was manipulating
    files that only exist on the cloud and not on my local disk because I have no local disk space. However,
    on performing a rename of a file that only exists remotely, I'd get a spurious error such as

    $ drive rename WINTER2017/h1.s binary.s rename /Users/emmanuelodeke/[email protected]/ENGINEERING/WINTER2017/h1.s /Users/emmanuelodeke/[email protected]/ENGINEERING/WINTER2017/binary.s: no such file or directory
    

    ๐Ÿ›  This issue was fixed by PR #827.

    ๐Ÿ–จ introduced a debug print mode.
    This feature was implemented in PR #830.
    Using environment flag DRIVE_DEBUG=true (carefully chosen to avoid clashing with
    other programs that might rely on DEBUG), one can get debug information dumped to standard output.
    The main purpose of this feature is to aid in easy debugging and particularly to alleviate
    #829 in which the reporter could have been informed ASAP
    that their directory had a .gd directory instead of them having to use their mental microscope
    to figure out what went wrong.

    Typical output will look like

    $ DEBUG=true drive list share-testing/ [/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:main.discoverContext:1857] contextPath: /Users/emmanuelodeke/[email protected]/share-testing [/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/cmd/drive/main.go:main.discoverContext:1867] driveRoot: "/Users/emmanuelodeke/[email protected]" relToRoot: ""[/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/rc.go:github.com/odeke-em/drive/src.ResourceMappings:92] RCPath: /Users/emmanuelodeke/[email protected]/share-testing/.driverc [/Users/emmanuelodeke/go/src/github.com/odeke-em/drive/src/rc.go:github.com/odeke-em/drive/src.ResourceMappings:108] parsedContent from"/Users/emmanuelodeke/[email protected]/share-testing/.driverc"{"global": { "depth": -1 } } [Commands.List] #0 "/share-testing"-- owner 175.00B 0Bwu8laYc9RTPa28zVk9Td2hTVWc 2016-06-30 15:32:25 +0000 UTC /share-testing/SciqPCKrhi.go -- owner 309.00B 0Bwu8laYc9RTPTXRYblNqQXBSQzQ 2016-02-03 08:12:15 +0000 UTC /share-testing/outf.go -s owner 39.70KB 0Bwu8laYc9RTPOVNSeElpdFBpS2M 2012-02-02 12:00:00 +0000 UTC /share-testing/ComedyPunchlineDrumSound.mp3
    

    Binaries

    $ make
    CGO_ENABLED=0 GOOS=linux GOARM=5 GOARCH=arm go build -o ./bin/drive_armv5 ./cmd/drive
    CGO_ENABLED=0 GOOS=linux GOARM=6 GOARCH=arm go build -o ./bin/drive_armv6 ./cmd/drive
    CGO_ENABLED=0 GOOS=linux GOARM=7 GOARCH=arm go build -o ./bin/drive_armv7 ./cmd/drive
    CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o ./bin/drive_armv8 ./cmd/drive
    CGO_ENABLED=0 GOOS=darwin go build -o ./bin/drive_darwin ./cmd/drive
    CGO_ENABLED=0 GOOS=linux go build -o ./bin/drive_linux ./cmd/drive
    rm -f ./bin/md5Sums.txt
    find ./bin -type f -name "drive_*" -exec md5 {} >> ./bin/md5Sums.txt \;
    cat ./bin/md5Sums.txt
    MD5 (./bin/drive_armv5) = 990d899bdff2e54cd401ece624a4f8be
    MD5 (./bin/drive_armv6) = 812689b6bb2113948c8394854bbde47f
    MD5 (./bin/drive_armv7) = 222be8926c35353f81baba014d926c71
    MD5 (./bin/drive_armv8) = ecc56e8df5af018eab40097d261e862d
    MD5 (./bin/drive_darwin) = 4b2528c13a5fb955bf28f48beec197ca
    MD5 (./bin/drive_linux) = a192d38419377aa27a45d90055c56a14
    

    ๐Ÿš€ This release as mentioned is the first of a new year 2017! Hoping for brighter days and more stability, and more user engagement. Our customers, the drive community deserve to have the best commandline client
    for Google Drive and that's a constant that we all strive for!
    Thank you very much everyone for your patience, contributions, analysis, encouragement, critic, ideas etc.

    In particular, I'd like to give a shout to @jean-christophe-manciot for finding many bugs and for all the feedback!

    This is a dawn of a new era, let's go for it!! Happy New Year!
    ๐Ÿฑ obamabiden


Previous changes from v0.3.8

  • ๐Ÿš€ This release is the end of the summer 2016 release -- a return from a 3 month hiatus, in which a whole lot of bugs were fixed.

    It features:

    • ๐Ÿ— Travis install fix. After adding a Makefile, the build was broken. PR #666 by @sselph fixed this bug
    • share, --with-link to avoid public file indexing, but only those with a link can access it.
      This feature was requested for in #568 by @RickCogley and the fix was guided by @thebaddie. The problem was that doing

      $ drive pub afile

    would make Google publicly index the shared file, of which for private files shouldn't be indexed. The fix for it involved ensuring that property WithLink. It was implemented in PR #674, thus you can now do

    $ drive share --with-link release-videos/stewie\_dez\_nuts.mp4 Provide access for accountType(s) user anyone For roles(s) reader Addressees: + Anyone with the link File(s) to share: + stewie\_dez\_nuts.mp4 Proceed with the changes? [Y/n]:y successful share for stewie\_dez\_nuts.mp4 with email "", role "reader" accountType "anyone"$ drive url release-videos/stewie\_dez\_nuts.mp4 /release-videos/stewie\_dez\_nuts.mp4: https://drive.google.com/open?id=0By5qQkvRAeV2dHhVaUpiUzFibm8
    
    • ๐Ÿ›  Fixed a spurious error when parsing .driverc files, that was caused by function signatures getting changed but variable reuse meant an old error was used. This bug was reported by @dconathan in #677, then duplicate reported by @staltux in #679. It was fixed by PR #678.
    • โž• Added the ability to type select during pull/push ie folders only, files only etc. This feature was requested by @brandoncurtis in #670. It was fixed by #673.
      You can now do

      $ drive push --directories a1/m1 $ drive pull --files tf1/tf2

    • ๐Ÿ›  Fixed a nil/uninitialized filter dereference. This bug was caused by PR #632. It was reported by @Sjlver in #681. It was fixed by #682.

    • ๐Ÿ”ง Encryption and decryption key variables can now be recognized in the .driverc configuration file. This bug was reported by @quisar in #683. The bug was that the variables for encryption-key and decryption-key were not recognized. It was fixed by #684

      $ cat .driverc hidden=true encryption-password=bonjour decryption-password=bonjour $ drive push test.mp4 # Should be encrypted at rest$ drive pull --piped test.mp4 > outf && diff outf test.mp4 # Should be decrypted properly

    • ๐Ÿ›  Fixed up --no-clobber description for push. This bug was reported by @kcwu in #685. It was fixed by #688.

    • โšก๏ธ Updated the QR code dispensing drive-server to write a png image using the response writer's .Write method instead of going first through fmt.Fprintf(res, "%s", pngImage), the code is clearer and faster. This update was made in #689.

    • pull/export can now export to the same directory and level. This feature was requested by @AtcR in #660. The use case is to avoid creating too many exports' directories and to allow your exported files to be placed in one place. It was added in #696.

      $ drive pull --explicitly-export --exports-dir ~/Desktop/exp --export pdf,txt,odt --same-exports-dir Resolving... + /test-exports/few.docs + /test-exports/few + /test-exports/influx Addition count 3 Proceed with the changes? [Y/n]:y Exported '/Users/emmanuelodeke/[email protected]/test-exports/influx' to '/Users/emmanuelodeke/Desktop/exp/influx.pdf'Exported '/Users/emmanuelodeke/[email protected]/test-exports/influx' to '/Users/emmanuelodeke/Desktop/exp/influx.txt'Exported '/Users/emmanuelodeke/[email protected]/test-exports/few' to '/Users/emmanuelodeke/Desktop/exp/few.pdf'Exported '/Users/emmanuelodeke/[email protected]/test-exports/few.docs' to '/Users/emmanuelodeke/Desktop/exp/few.docs.txt'Exported '/Users/emmanuelodeke/[email protected]/test-exports/few.docs' to '/Users/emmanuelodeke/Desktop/exp/few.docs.odt'Exported '/Users/emmanuelodeke/[email protected]/test-exports/few.docs' to '/Users/emmanuelodeke/Desktop/exp/few.docs.pdf'

    • ๐Ÿง Disable the automatic creation of *.desktop files on Linux. This feature was requested by @chevillet in #697. It was added in #698 and now you can do

      $ drive pull --desktop-links=false

    and you could keep this behavior turned on in your .driverc by

    $ cat \<\< ! \>\> .driverc\> desktop-links=false\> !
    
    • ๐Ÿ›  list --trashed now returns all items in the same current working directory. This bug was reported by @njbbaer in #695. It was also fixed by @njbbaer in #700. The diagnosis of the bug was two fold which you can find by reading #700 (comment).
    • ๐Ÿ“š --ignore-checksum documentation was made more coherent. This was made by @gjalsem in #703.
    • audited open file descriptors to ensure that they closed to avoid running out of open files ie descriptor leakage. This issue was reported in #711 by a quick audit

      $ git grep -E 'os.Open(' src/* config/* drive-gen/*.go src/misc.go: f, fErr := os.Open(p) src/remote.go: file, err := os.Open(fsAbsPath) src/types.go: fh, err := os.Open(f.BlobAt)

    in which I noticed some opens but no accompanying closes.
    ๐ŸŽ Alas there was a call site that leaked its descriptors ie during file upload. This behavior was fixed by #712. The fix involved avoiding unnecessary opens but also attaching a file.Close handler to clean up after the caller returned. That change results in better performance and less resource exhaustion.

    • ๐Ÿ›  Fixed push mounted(push -m) that always unloaded entries in the root as reported in #641.
      ๐Ÿ›  This bug was a regression from #616. It was fixed by #713. The fix also ensured that we caught erraneous proceedings after push mounted had returned due to a failure to return when done.
    • โšก๏ธ list --sort now accepts multiple sort properties. This bug was reported in #714. It was fixed by #715 that ensures that sorters can be passed in delimited by comma. It also updated the doc on list sorting. So now we can perform composite sorts for example by name in descending order, then modification time in ascending:
      order:

      $ drive list -sort name_r,modtime

    • ๐Ÿ“ฆ Fedora package for drive. This was contributed by @agrez in #717. Please see https://github.com/odeke-em/drive/blob/master/platform_packages.md#fedora for how to get drive on Fedora.

    • ๐Ÿ›  return a non-zero error after fixing clashes so that scripts can refresh and then re-pull. This was reported by @yottabit42 in #720. Given that a call for fixing clashes during pull or push stops the operation, it is necessary that the outside world be relayed to, that the operation can be retried, e.g by a shell script. This behavior was fixed by @sselph in #722.

    • touch now allows custom times to be set. This feature was requested by @bjohas in #726. It was implemented in #729.
      Now you can set the desired times on your remote files instead of them just always being at the time that drive touch was performed on the remote server. You can do pass in custom times, and custom time formats for example:

      $ drive touch --time 20120202120000 ComedyPunchlineDrumSound.mp3 /share-testing/ComedyPunchlineDrumSound.mp3: 2012-02-02 12:00:00 +0000 UTC $ $ drive touch --format "2006-01-02-15:04:05.0000Z" --time "2016-02-03-08:12:15.0070Z" outf.go /share-testing/outf.go: 2016-02-03 08:12:15 +0000 UTC $ $ drive touch --duration -30h ComedyPunchlineDrumSound.mp3 outf.go /share-testing/outf.go: 2016-09-10 08:06:39 +0000 UTC /share-testing/ComedyPunchlineDrumSound.mp3: 2016-09-10 08:06:39 +0000 UTC

    • ๐Ÿ›  Fixed pulling from a deprecated Google Web Hosting URL. This bug was a notice from Google Drive https://support.google.com/drive/answer/2881970?hl=en. However it was reported by @dermesser in #734 (comment) and revealed by @cdown's bug report in #734. It was fixed by @dermesser in #735. It was then updated by #736.

    • ๐Ÿ›  Fixed unrelayed errors on pulls. This bug was reported by @cdown in #734. It was fixed by #737.

    • ๐Ÿ›  Documented lone modification time change handling during checksum verification. This request for clarity was made by @tkeith in #690. It was fixed by #742.

    • Propagating and handling errors from remote file resolution/pagination. This bug was catastrophic and had plagued drive for a full year and more. The problem was in the design of the pagination where the paginator asynchronously retrieved files and only returned a channel of files. The problem was that during resolving files, we would encounter a remote Google Drive error e.g from its servers crashing, or from quota/rate-limit exhaustion. Not propagating the encountered error meant that during a pull or push, the remote side would be reported to not have files which would cause a full upload/deletion. This bug was reported in very many places; to name a few:

    • @CountryBumkin reported #480. #480 is probably the most commented on issue in drive because a rate limiting bug on Google Drive's backend(which may still exist as of writing) on drive's globally used quota caused spurious deletions and reuploads for users all around the world.

    • @brandoncurtis reported #668 in which intermittent 503 errors from Google Drive permitted pushes(because the 503 was never propagated to the caller) and then caused spurious conflicts so the user would be stuck in a state of limbo.

    • @jimr6007 reported #728 in which duplicate files were being pushed upstream because a previous 500 intermittent error on Google Drive would cause the remote files to be reported as non-existent.

    • @creighto reported #738 in which the opposite of @jimr6007's case ie a pull instead of a push, caused a spurious need for deletions.

    Binaries

    $ make CGO\_ENABLED=0 GOOS=linux GOARM=5 GOARCH=arm go build -o ./bin/drive\_armv5 ./cmd/drive CGO\_ENABLED=0 GOOS=linux GOARM=6 GOARCH=arm go build -o ./bin/drive\_armv6 ./cmd/drive CGO\_ENABLED=0 GOOS=linux GOARM=7 GOARCH=arm go build -o ./bin/drive\_armv7 ./cmd/drive CGO\_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o ./bin/drive\_armv8 ./cmd/drive CGO\_ENABLED=0 GOOS=darwin go build -o ./bin/drive\_darwin ./cmd/drive CGO\_ENABLED=0 GOOS=linux go build -o ./bin/drive\_linux ./cmd/drive rm -f ./bin/md5Sums.txt find ./bin -type f -name "drive\_\*" -exec md5 {} \>\> ./bin/md5Sums.txt \;cat ./bin/md5Sums.txt MD5 (./bin/drive\_armv5) = 56521b19ec572f27b4fc434e987e1c10 MD5 (./bin/drive\_armv6) = 2dfba2e2f4d40fe4987f0f641ddaaccb MD5 (./bin/drive\_armv7) = 831790fc2e66a033cee20b5bd6b03695 MD5 (./bin/drive\_armv8) = 894901be67472b1fe49c4acc962cdc8b MD5 (./bin/drive\_darwin) = 6e9968de26e93448eedc9835b66ef5fb MD5 (./bin/drive\_linux) = e175943bc56a8f01b7c7574ddf383986
    

    ๐Ÿš€ This release is very interesting because it is a deal breaker for a lot of reliability and trust that users can now get in drive. My biggest apologies to all the users that were affected by lack of error propagation during remote resolution/pagination. Unfortunately I hadn't had much time to work on it, but this afternoon felt right.

    ๐Ÿš€ This was the end of the summer release and reflecting back on the lessons learned while working on drive in my spare time, getting help from numerous volunteers and well wishers, interacting with the users both on the issue tracker and via email; I have to say this is a great experience. Fighting these fires has given me wisdom and experience to keep working on other things.

    ๐Ÿš€ With this release, I hope this takes us to new frontiers establishing drive as a product that users can rely on with reliability, speed, performance and ease of use with all the features contained in here.

    ๐Ÿš€ I am looking forward to the next release!

    The Summer has ended but have a great Fall season and keep filing those bugs, tell your friends about drive and enjoy!

    ๐Ÿฑ obamamicdrop

    ๐Ÿฑ lastcall