Snowflake cloud data platform 如何使用Snowpipe ingestFile SDK摄取csv文件？

我试图使用ingest sdk在我的Java程序中摄取一个文件，但出现了一个错误。我已经提前创建了桌子、舞台和管道。我还将我的CSV文件放入了内部阶段，以准备接收。现在我的Java程序只需要调用ingestFiles来加载文件。然而，当我打电话时，我得到一个404错误（请参阅下面附带的日志文件和Java程序）

我已使用以下步骤创建了我的公钥和私钥：

还请注意，通过手动发出与管道关联的COPY命令，可以成功加载文件。但是，我更喜欢使用ingest SDK，以便更好地捕获响应

我怀疑这是一个许可问题，但不确定。任何帮助都将不胜感激

03:11:06.275[main]INFO com.pardi.snowpipetest.instetest-已成功从文件keys/rsa_key.p8加载私钥

[main]WARN net.snowflake.inset.connection.RequestBuilder-无法读取版本信息：java.nio.file.FileSystemNotFoundException

[main]INFO net.snowflake.insect.connection.SecurityManager-创建主题为WX11111.JPARDI的令牌

[main]INFO net.snowflake.insect.connection.SecurityManager-使用颁发者WX11111.JPARDI.SHA256创建令牌：xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[main]INFO net.snowflake.insect.connection.SecurityManager-已创建新的JWT-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[main]INFO net.snowflake.insect.connection.RequestBuilder-创建带参数的RequestBuilder：帐户：WX11111，用户：JPARDI，方案：https，主机：.us-east-1.snowflakecomputing.com，端口：443

[main]INFO net.snowflake.insect.SimpleIngestManager-发送请求UUID-

[main]INFO net.snowflake.inset.connection.RequestBuilder-创建的插入请求：https://.us-east-1.snowflakecomputing.com:443/v1/data/pipes/DEMO_DB.PUBLIC.COVETRUS_SNOWFLAKE_BATCH_SINK_SNOWPIPETEST_PIPE_H_CLIENT_0/insertFiles?requestId=39a2c1e4-c637-4424-bb68-42c4eb71873a

[main]INFO net.snowflake.insect.SimpleIngestManager-尝试取消对插入响应的编组-未找到HttpResponseProxy{HTTP/1.1 404[内容类型：application/json，日期：周二，2020年2月25日08:11:10 GMT，服务器：nginx，严格传输安全：max age=31536000，X-Content-Type-Options:nosniff，X-Frame-Options:deny，Connection:keep-alive]net.snowflake.insect.internal.apache.http.client.entity。DecompressingEntity@29c80149}

[main]WARN net.snowflake.ingest.connection.ServiceResponseHandler-在unmarshallInsert响应-404中找到异常状态代码

[main]错误net.snowflake.ingest.connection.ServiceResponseHandler-在服务响应中找到状态代码404

03:11:11.943[主]信息com.pardi.snowpipetest.IngestTest-服务异常：

HTTP状态：404

{

消息：指定的对象不存在或未授权。未找到管道

数据：空

}

package com.pardi.snowpipetest；
导入net.snowflake.insect.SimpleIngestManager；
导入net.snowflake.insect.connection.HistoryResponse；
导入net.snowflake.ingest.connection.IngestResponse；
导入net.snowflake.ingest.connection.IngestResponseException；
导入org.bouncycastle.asn1.pkcs.PrivateKeyInfo；
导入org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider；
导入org.bouncycastle.openssl.PEMParser；
导入org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter；
导入org.bouncycastle.openssl.jcajce.JceOpenSSLPKCS8DecryptorProviderBuilder；
导入org.bouncycastle.operator.InputDecryptorProvider；
导入org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo；
导入org.slf4j.Logger；
导入org.slf4j.LoggerFactory；
导入org.springframework.core.io.ClassPathResource；
导入java.io.*；
导入java.security.PrivateKey；
导入java.security.security；
导入java.util.Set；
导入java.util.TreeSet；
导入java.util.concurrent.*；
公共类测试{
私有静态最终记录器Logger=LoggerFactory.getLogger（IngestTest.class）；
私有静态字符串主机=“.us-east-1.snowflakecomputing.com”；
私有静态字符串帐户=”；
私有静态字符串user=“xxxxxx”；
私有静态字符串密码短语=“xxxxxxxxx”；
专用静态int端口=443；
私有静态字符串数据库=“DEMO_DB”；
私有静态字符串schema=“PUBLIC”；
私有静态字符串管道=“COVETRUS\u雪花\u批量\u水槽\u雪花管道测试\u管道\u H\u客户端\u 0”；
私有静态字符串fqPipe=数据库+“+”+架构+“+”+管道；
私有静态私钥私钥；
专用静态SimpleIngestManager；
公共静态void main（字符串[]args）引发异常{
字符串privateKeyStr=loadPrivateKey（）；
privateKey=parseEncryptedPrivateKey（privateKeyStr，密码短语）；
manager=新的SimpleIngestManager（帐户、用户、fqPipe、私钥、“https”、主机、端口）；
Set files=new TreeSet（）；
添加（“covetrus\u snowflake\u batch\u sink\u snowpipetest\u stage\u h_client/h_client.csv”）；
试一试{
IngestResponse-response=manager.ingestFiles（manager.wrapFilePath（files），null）；
LOGGER.info（“response=“+response.toString（））；
HistoryResponse history=waitForFilesHistory（文件）；
LOGGER.info（“收到的历史记录响应：+history.toString（））；
}捕获（摄入响应异常e）{
info（“服务异常：+e.toString（））；
}捕获（例外e）{
LOGGER.info（“异常：+e.getMessage（））；
}
}
私有静态字符串loadPrivateKey（）引发IOException{
字节[]键字节；
字符串filename=“keys/rsa_key.p8”；
File privateKeyFile=null；
试一试{
privateKeyFile=新类路径资源（文件名）.getFile（）；
文件输入
package com.pardi.snowpipetest;

import net.snowflake.ingest.SimpleIngestManager;
import net.snowflake.ingest.connection.HistoryResponse;
import net.snowflake.ingest.connection.IngestResponse;
import net.snowflake.ingest.connection.IngestResponseException;
import org.bouncycastle.asn1.pkcs.PrivateKeyInfo;
import org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider;
import org.bouncycastle.openssl.PEMParser;
import org.bouncycastle.openssl.jcajce.JcaPEMKeyConverter;
import org.bouncycastle.openssl.jcajce.JceOpenSSLPKCS8DecryptorProviderBuilder;
import org.bouncycastle.operator.InputDecryptorProvider;
import org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.io.ClassPathResource;

import java.io.*;
import java.security.PrivateKey;
import java.security.Security;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.*;

public class IngestTest {
    private static final Logger LOGGER = LoggerFactory.getLogger(IngestTest.class);
    private static String host = "<xxxx>.us-east-1.snowflakecomputing.com";
    private static String account = "<xxxxx>";
    private static String user = "xxxxxx";
    private static String passPhrase = "xxxxxxxxx";
    private static int port = 443;
    private static String database = "DEMO_DB";
    private static String schema = "PUBLIC";
    private static String pipe = "COVETRUS_SNOWFLAKE_BATCH_SINK_SNOWPIPETEST_PIPE_H_CLIENT_0";
    private static String fqPipe = database + "." + schema + "." + pipe;

    private static PrivateKey privateKey;
    private static SimpleIngestManager manager;

    public static void main(String[] args) throws Exception {
        String privateKeyStr = loadPrivateKey();
        privateKey = parseEncryptedPrivateKey(privateKeyStr, passPhrase);

        manager = new SimpleIngestManager(account, user, fqPipe, privateKey, "https", host, port);

        Set<String> files = new TreeSet<>();
        files.add("covetrus_snowflake_batch_sink_snowpipetest_stage_h_client/h_client.csv");

        try {
            IngestResponse response = manager.ingestFiles(manager.wrapFilepaths(files), null);
            LOGGER.info("response=" + response.toString());

            HistoryResponse history = waitForFilesHistory(files);
            LOGGER.info("Received history response: " + history.toString());
        } catch (IngestResponseException e) {
            LOGGER.info("Service exception: " + e.toString());
        } catch (Exception e) {
            LOGGER.info("Exception: " + e.getMessage());
        }
    }

    private static String loadPrivateKey() throws IOException {
        byte[] keyBytes;

        String filename = "keys/rsa_key.p8";

        File privateKeyFile = null;

        try {
            privateKeyFile = new ClassPathResource(filename).getFile();
            FileInputStream fis = new FileInputStream(privateKeyFile);
            DataInputStream dis = new DataInputStream(fis);
            keyBytes = new byte[(int) privateKeyFile.length()];
            dis.readFully(keyBytes);
            dis.close();
        } catch (IOException e) {
            LOGGER.info("FATAL: error loading private key from file " + filename + ", exception=" + e.getMessage());
            e.printStackTrace();
            throw e;
        }

        String privateKeyStr = new String(keyBytes);
        LOGGER.info("successfully loaded private key from file " + filename);
        return privateKeyStr;
    }

    public static PrivateKey parseEncryptedPrivateKey(String key, String passphrase) {
        Security.addProvider(new BouncyCastleFipsProvider());
        try {
            PEMParser pemParser = new PEMParser(new StringReader(key));
            PKCS8EncryptedPrivateKeyInfo encryptedPrivateKeyInfo = (PKCS8EncryptedPrivateKeyInfo) pemParser.readObject();
            pemParser.close();
            InputDecryptorProvider pkcs8Prov = new JceOpenSSLPKCS8DecryptorProviderBuilder().build(passphrase.toCharArray());
            JcaPEMKeyConverter converter = new JcaPEMKeyConverter().setProvider(BouncyCastleFipsProvider.PROVIDER_NAME);
            PrivateKeyInfo decryptedPrivateKeyInfo = encryptedPrivateKeyInfo.decryptPrivateKeyInfo(pkcs8Prov);
            return converter.getPrivateKey(decryptedPrivateKeyInfo);
        } catch (Exception e) {
            throw new RuntimeException("Invalid encrypted private key or passphrase");
        }
    }

    private static HistoryResponse waitForFilesHistory(Set<String> files)
            throws Exception {
        ExecutorService service = Executors.newSingleThreadExecutor();

        class GetHistory implements
                Callable<HistoryResponse> {
            private Set<String> filesWatchList;

            GetHistory(Set<String> files) {
                this.filesWatchList = files;
            }

            String beginMark = null;

            public HistoryResponse call()
                    throws Exception {
                HistoryResponse filesHistory = null;
                while (true) {
                    Thread.sleep(500);
                    HistoryResponse response = manager.getHistory(null, null, beginMark);
                    if (response.getNextBeginMark() != null) {
                        beginMark = response.getNextBeginMark();
                    }
                    if (response != null && response.files != null) {
                        for (HistoryResponse.FileEntry entry : response.files) {
                            //if we have a complete file that we've
                            // loaded with the same name..
                            String filename = entry.getPath();
                            if (entry.getPath() != null && entry.isComplete() &&
                                    filesWatchList.contains(filename)) {
                                if (filesHistory == null) {
                                    filesHistory = new HistoryResponse();
                                    filesHistory.setPipe(response.getPipe());
                                }
                                filesHistory.files.add(entry);
                                filesWatchList.remove(filename);
                                //we can return true!
                                if (filesWatchList.isEmpty()) {
                                    return filesHistory;
                                }
                            }
                        }
                    }
                }
            }
        }

        GetHistory historyCaller = new GetHistory(files);
        //fork off waiting for a load to the service
        Future<HistoryResponse> result = service.submit(historyCaller);

        HistoryResponse response = result.get(2, TimeUnit.MINUTES);
        return response;
    }
}