Fix git-lfs smudge error: Error downloading file, object does not exist on server

What to do when git's smudge filter lfs fails

Posted by Orville Bennett on 2 April 2021
Read time: about 5 minutes

I ran into this problem recently, when attempting to migrate a repository (repo) from my personal gitea instance to github. It started innocently enough. I had a git repository and wanted to test parallel test runs with cypress. The CI provider I was using (drone), did not support this feature and, after speaking with the cypress folks, they suggested trying with github actions. Sure. Why not. Should be easy. Little did I know this would lead me down a git-lfs sized rabbit hole.

So, first things first, I performed the git repo migration incorrectly. I forgot all about the git-lfs files that were in this repo, so my migration looked something like this.

  1. Modify existing repo's .git/config file to replace origin and point to github.
  2. Push to github
  3. Done

The problem with doings things this way is that there are things that need to happen to push the lfs stored objects to their new location that did not happen.

When cloning, git is not set up—by default—to automatically download files from the lfs location. Everything seemed fine when I cloned the repo from the new location, like so:

Cloning into 'inspiringhopechurch.com.g'...
remote: Enumerating objects: 1748, done.
remote: Counting objects: 100% (1748/1748), done.
remote: Compressing objects: 100% (519/519), done.
remote: Total 1748 (delta 1210), reused 1727 (delta 1197), pack-reused 0
Receiving objects: 100% (1748/1748), 5.48 MiB | 5.33 MiB/s, done.
Resolving deltas: 100% (1210/1210), done.

But it should have looked more like so:

Cloning into 'inspiringhopechurch.com.g'...
remote: Enumerating objects: 1748, done.
remote: Counting objects: 100% (1748/1748), done.
remote: Compressing objects: 100% (519/519), done.
remote: Total 1748 (delta 1210), reused 1727 (delta 1197), pack-reused 0
Receiving objects: 100% (1748/1748), 5.48 MiB | 4.99 MiB/s, done.
Resolving deltas: 100% (1210/1210), done.
Filtering content: 100% (12/12), 47.80 MiB | 7.65 MiB/s, done.

If we have already cloned, we can use the git lfs command within the repo to pull the lfs files with git lfs pull. Even with these options there are still problems. We still haven't pushed to the new repo's lfs storage yet. So a git lfs pull will give the following error:

[7bc8118e51aac96a07f0123b5dfa8325a9d7404fa998d63776f08a9cc8237c0c] 
Object does not exist on the server: [404] Object does not exist on the server
error: failed to fetch some objects from 'https://github.com/user/repo.git/info/lfs

To make git pull from lfs storage while cloning we can add some options to ~/.gitconfig, e.g.:

[filter "lfs"]
	clean = git-lfs clean -- %f
	smudge = git-lfs smudge -- %f
	process = git-lfs filter-process
	required = true

But if we do that then (since the repo still doesn't have the lfs files) you'll get a smudge error during cloning:

Cloning into 'repo'...
remote: Enumerating objects: 1736, done.
remote: Counting objects: 100% (1736/1736), done.
remote: Compressing objects: 100% (511/511), done.
remote: Total 1736 (delta 1202), reused 1719 (delta 1193), pack-reused 0
Receiving objects: 100% (1736/1736), 5.48 MiB | 6.79 MiB/s, done.
Resolving deltas: 100% (1202/1202), done.
Downloading static/assets/webfont.woff (14 KB)
Error downloading object: static/assets/webfont.woff (7d3a21b): Smudge error: Error downloading static/assets/webfont.woff (7d3a21b4869056b06276f51018070848c876d86570c56f9fdf726d8fed53ae9d): [7d3a21b4869056b06276f51018070848c876d86570c56f9fdf726d8fed53ae9d] Object does not exist on the server: [404] Object does not exist on the server

You could try to restore from git but then you'd get the error:

git restore --source=HEAD :/
Downloading static/assets/webfont.woff (14 KB)
Error downloading object: static/assets/webfont.woff (7d3a21b): Smudge error: Error downloading static/assets/webfont.woff (7d3a21b4869056b06276f51018070848c876d86570c56f9fdf726d8fed53ae9d): [7d3a21b4869056b06276f51018070848c876d86570c56f9fdf726d8fed53ae9d] Object does not exist on the server: [404] Object does not exist on the server

Errors logged to ~/Source/repo/.git/lfs/logs/20210223T220123.881257.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: static/assets/webfont.woff: smudge filter lfs failed

OK, I'll stop talking about all the possible errors. Hopefully by now you've picked up on what the solution is. Push the files using git lfs to the new repo. How does one do that?

git lfs push --all git@github.com:user/repo.git

This was done in the original repo where I modified the .git/config file to point to the new origin.

There you have it, all that exposition for a one line solution. Hopefully, the long explanations help you (and future me) understand the problem and make it clear why the solution is as simple as it is. It took me a while to get to that point, but now I have a better understanding of what git lfs does, and more importantly, what I should (not) do when migrating future git repos.

So here's the right way to migrate git repos with lfs files in 2021.

git push --mirror git@github.com:user/repo.git
git lfs push --all git@github.com:user/repo.git