Files uploaded, at google/im drive but not in Plex


#1

Hello,

I noticed that there are some movies that are not displayed in my Plex library. Although they were uploaded normally and are also displayed on Google and in the drive in the operating system. Additionally, I was shown this message when uploading other files, i.e. not the same ones as mentioned.

2018/10/12 18:02:14 NOTICE: Crimson Tide - In tiefster Gefahr (1995): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: Der Vagabund und das Kind (1921): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: No Way Out - Gegen die Flammen (2017): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: Papillon (1973): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: Starship Troopers (1997): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: The Avengers (2012): Duplicate directory found in destination - ignoring
2018/10/12 18:02:14 NOTICE: The Help (2011): Duplicate directory found in destination - ignoring

screenshot from google web:

Appear to be duplicated. But still they are not displayed in Plex? What can I do? I always moved with the same command and never changed anything in the config or with the password.

move command:

rclone move -v /Volumes/backup/Sync secretG:/Movies

mount command:

rclone mount --allow-other --allow-non-empty --cache-db-purge -v --log-file="/home/hd1/ttopxelp/rclonelogs/rclone.log" secretG: /home/hd1/ttopxelp/cloud-media/ &

Is there any way to find out if this still affects more files? Library is a bit bigger, about 35 TB

Hope you have an idea, thank you

edit: added screenshot


#2

If you look at the mount and a ls in the directory, do you see the files?


#3

the folder with the subtitles in it is displayed when I do rclone ls. The other one with the movie file is not. The funny thing is that the other folder on Google has your same name in the browser.


#4

It is possible to have files and folders with the same name on Google drive. This usually only happens when multiple copy or move commands are run in parallel.

You should look at rclone dedupe to remove these duplicate entries. After that all entries should show as expected.


#5

Is there a way to see how long dedupe takes?

I used the following command:

rclone dedupe -v --dedupe-mode interactive --tpslimit 1 secretG:/Movies

2018/10/16 15:05:39 INFO : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2018/10/16 15:06:18 INFO : Encrypted drive ‘secretG:/Movies’: Looking for duplicates using interactive mode.

I only took the Movies folder, because the movies were displayed there.

After a while I got this message now

2018/10/16 16:59:37 Failed to dedupe: Encrypted drive ‘secretG:/Movies’: can’t merge directories


#6

There is no way to see the progress.
But you can use --fast-list to speed it up.

Is there a particular reason why you set --tpslimit 1?

Unfortunately it is currently not possible to dedupe a crypt remote.

There is a pending issue on GitHub which I thought was already solved.

But you can dedupe you plain Google Drive:

# Get the encrypted Movies name
rclone cryptdecode --reverse secretG: Movies
# Use the encrypted name on the regular remote
rclone dedupe -v --fast-list --dedupe-mode=skip --tpslimit=1 normalG:8mrvughitnvnjo10ae9co

--dedupe-mode=skip will only merge folders which contain unique filenames (which sounds like your use case). This could be enough to fix your problem.
If this doesn’t help, you will have to dedupe encrypted filenames by hand with --dedupe-mode=interactive :roll_eyes:


#7

Had read somewhere that it might help to reduce API hits

Sorry about the stupid question, but how do I do that now?^^


#8

Using --tpslimit will just slow down dedupe and reserve API calls/s for other rclone commands like mount or copy. If you want dedupe to complete as fast as possible you can remove this flag.

No problem, I will try to explain it :grinning:

If you only want to dedupe you Movies folder from the encrypted remote on your not encrypted remote, you need to find the encrypted name of this folder. The easiest way is to use cryptdecode:

% rclone cryptdecode --reverse secretG: Movies
Movies 	 8mrvughitnvnjo10ae9co

You can then use this encrypted name to dedupe the folder. Combine the value of the remote = setting of the secretG section in your rclone.conf with the encypted folder name:

rclone dedupe -v --fast-list --dedupe-mode=skip gdrive:crypt/8mrvughitnvnjo10ae9co

If you just got duplicate folders and not duplicate files, --dedupe-mode=skip should be the right flag to deduplicate these folders into a single one.
When this is not enough and there are duplicate files, you can use --dedupe-mode=interactive (or some other mode) to choose how to deduplicate these files.


#9

thanks for your help. But unfortunately its not working. This is what I tried…

these are my remotes

Current remotes:

Name Type
==== ====
Google drive
gcache cache
secretG crypt

my command:

rclone dedupe -v --fast-list --dedupe-mode=interactive secretG:crypt/Movies.bin

Error:

2018/10/17 09:04:57 Failed to dedupe: find duplicate dirs: directory not found


#10

You mixed up the remote names of the encrypted and not encrypted remote.

Your encrypted remote is secretG: containing the Moviesfolder.
Your not encrypted (origin) remote is Google:.

In order to dedupe your not encrypted folder you need to call dedupe on the Google: remote.

rclone dedupe -v --fast-list --dedupe-mode=interactive Google:crypt/Movies

It seems like you disabled name encryption in your config, so the cryptdecode step from my last post is not needed. (also cryptdecode can’t differentiate between file and folder names, so it adds the .bin suffix to the folder which is incorrect)


#11

hm I already had tried that. :flushed:

message:

rclone dedupe -v --fast-list --dedupe-mode=interactive Google:crypt/Movies
2018/10/19 18:32:05 INFO : Google drive root ‘crypt/Movies’: Looking for duplicates using interactive mode.
2018/10/19 18:32:05 Failed to dedupe: find duplicate dirs: directory not found


#12

Can you post your rclone.conf wihout the access tokens and passwords? This would help to understand your remote setup.


#13

my config

[secretG]
type = crypt
remote = gcache:
filename_encryption = off
password =
password2 =

[Google]
type = drive
client_id =
client_secret =
token =

[gcache]
type = cache
remote = Google:Plex
plex_url = http://172.17.0.2:32400
plex_username =
plex_password =
plex_token =
chunk_size = 10M
info_age = 24h
chunk_total_size = 10G