sql >> Databasteknik >  >> RDS >> PostgreSQL

Sqoop på Hadoop:NoSuchMethodError:com.google.common.base.Stopwatch.createStarted()

Efter lite ytterligare efterforskningar ser jag att Hadoop 2.x åsidosätter klassvägen, så lösningen är att skapa en uberjar och skicka det till hadoop.

Jag ändrade också till att använda den specifika sqoop-burken för hadoop260 istället.

Så jag skapade en pom.xml fil, körde maven package på den för att generera uberjar:

<?xml version="1.0" encoding="UTF-8"?>
<!-- SEE: https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
    <groupId>org.example.exporter</groupId>
    <artifactId>example-exporter-postgresql</artifactId>
    <version>0.0.1</version>
    <!-- YOUR_DEPENDENCIES -->
    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.sqoop/sqoop -->
        <dependency>
            <groupId>org.apache.sqoop</groupId>
            <artifactId>sqoop</artifactId>
            <version>1.4.7</version>
            <classifier>hadoop260</classifier>
        </dependency>
        <!-- https://mvnrepository.com/artifact/postgresql/postgresql -->
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>42.2.9</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.cloud.sql/postgres-socket-factory -->
        <dependency>
            <groupId>com.google.cloud.sql</groupId>
            <artifactId>postgres-socket-factory</artifactId>
            <version>1.0.15</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.avro/avro-tools -->
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-tools</artifactId>
            <version>1.9.1</version>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>org.apache.sqoop.Sqoop</mainClass>
                                </transformer>
                            </transformers>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/maven/**</exclude>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <relocations>
                                <relocation>
                                    <pattern>com</pattern>
                                    <shadedPattern>repackaged.com.google.common</shadedPattern>
                                    <includes>
                                        <include>com.google.common.**</include>
                                    </includes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>



  1. Få sista posten för varje månad i MySQL....?

  2. Postgresql, Begränsa uppdatering på specifika kolumner (skrivskyddad)

  3. SQLite Index

  4. Tidszon medveten datetime till sträng?