Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/linux/25.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181

Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/templates/2.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Linux 从不同网络目录间匹配的子目录下载所有csv文件_Linux_Csv_Sftp_Lftp - Fatal编程技术网

Linux 从不同网络目录间匹配的子目录下载所有csv文件

Linux 从不同网络目录间匹配的子目录下载所有csv文件,linux,csv,sftp,lftp,Linux,Csv,Sftp,Lftp,在每个月的第一天,我需要连接到SFTP服务器,并根据上个月的数据从某些子目录下载所有csv文件 要连接到的示例文件目录 sftp_url/csv/client1.1/10/ sftp_url/csv/client1.2/10/ sftp_url/csv/client1.3/10/ sftp_url/csv/client1.4/10/ sftp_url/csv/client2.1/10/ sftp_url/csv/client2.2/10/ sftp_url/csv/client2.3/10/ s

在每个月的第一天,我需要连接到SFTP服务器,并根据上个月的数据从某些子目录下载所有csv文件

要连接到的示例文件目录

sftp_url/csv/client1.1/10/
sftp_url/csv/client1.2/10/
sftp_url/csv/client1.3/10/
sftp_url/csv/client1.4/10/
sftp_url/csv/client2.1/10/
sftp_url/csv/client2.2/10/
sftp_url/csv/client2.3/10/
sftp_url/csv/client2.4/10/
子目录中的“10”表示月份,因此是“10月”。在/10/子目录中有多个csv文件,我需要将它们全部下载

  • 我已经找到了使用
    lftp
    连接到SFTP服务器的代码
  • 我有代码来确定“10”,即
    date-d“last month”+“%m”
但是,我无法找到如何定义我只想访问文件夹==上个月值的所有目录,而不列出所有完整的文件路径

有人能确认是否有一个简单的命令允许这种事情吗?如果这是非常直截了当的事情,我会道歉。我不熟悉命令行,这是一个陡峭的曲线。感谢您提供的任何帮助和反馈

#!/bin/sh

set -eu

SFTP_SERVER="sftp.location.com"
SFTP_USER="user"
SFTP_DIR="/csv"

DEST=$(basename $SFTP_DIR) ## defines directory based on where my current directory is and the sftp folder name

mkdir -p "$DEST" ## makes new directory based on basename and the month found in the downloaded folder

lftp sftp://$SFTP_USER@$SFTP_SERVER:$SFTP_DIR/ -e "lcd '$DEST'; mirror; bye"

ACCT_BREAKDOWNS=$(find "$DEST" -mindepth 1 -maxdepth 1 -type d) ## defines the month folders based on going 1 folder in from the sftp folder name
YEAR=$(TZ=UTC-24 date +%Y -d "-1 month") ## defines the year for the folders created, ensuring that it creates a new folder when we enter the new year
for ACCT_BREAKDOWN in $ACCT_BREAKDOWNS; do
        ACCT=$(basename "$ACCT_BREAKDOWN" | sed -E -e 's/_(.2|.1|.3)$//') ## strips folders to just show client name

        DATES=$(find "$ACCT_BREAKDOWN" -mindepth 1 -maxdepth 1 -type d -printf "%f\n") ## prints out a list of client names, with a separate one for each line
        #DATES="09 10" ## would be used to specify clear dates
        for DATE in $DATES; do
                mkdir -p "account/$ACCT/$YEAR-$DATE" ##create directory based on the client name after filename is stripped)

                for FILE in $(find "$ACCT_BREAKDOWN/$DATE" -type f); do ##  works on files that are in the sftp folder and date sub folder.
                        ln -f -t "account/$ACCT/$YEAR-$DATE" "$FILE" ## creates a hard link between the csv folder and account folder, placing files in a folder that match their client name and the right month.
                done
        done
done

rclone sync account/ remote:folder/folder ## syncs folder layout to gdrive

TODAY=$(date)
echo "$TODAY    global  script" > ~/error-log/log/log.tsv ## creates a tsv which is used a reference to what failed on my cronjobs

exit