大白糖奶兔的Blog
大白糖奶兔的Blog
初识Hadoop-SQOOP(Day13)

前戏

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_81ccf38226c1f188872378886ae93afc.jpg

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_92427ac52a35467315f826a876ea7222.jpg

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_39f10f712f6491b50ccd7bdf8bdba73a.jpg
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_075be1cf15f3158933066d613d014fbb.jpg

配置运行SQOOP

配置

中间格式:Json

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_b348b5937a1e1950198e1aa807a10811.jpg
tar -zxf sqoop-1.4.6-cdh5.7.6.tar.gz -C /opt/modules/

目录结构

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_5b9f0b23f3298ea8306e8551bc7f7693.jpg

赋值Jar包

[hadoop@hadoop sqoop-1.4.6-cdh5.7.6]$ cp /opt/tools/mysql-connector-java-5.1.27-bin.jar ./lib/
[hadoop@hadoop sqoop-1.4.6-cdh5.7.6]$ cp /opt/tools/java-json.jar ./lib/
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_51da29a15ff7e57149d3e7e482ad7cf4.jpg

修改配置文件

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_cfb8d4bc1e2bb0120be806d4dbbb6743.jpg
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_a20318ed362b5013dd4b5bc67469684e.jpg

运行测试

bin/sqoop list-tables --connect jdbc:mysql://hadoop:3306/metastore_db --username root --password Mysql_1234
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_e3ffbdc34b9b912f97556645b0084c22.jpg

使用SQOOP将MySQL转换为hfile

SQOOP是连接数据库和HDFS的工具
将数据库的数据转换为HDFS上的文件
Hive对HDFS上的文件进行分析,映射关系存在Resourcemanager的元数据中
SQOOP能够访问数据库的内容,也能读取元数据信息,将SQOOP拓展至数据库和Hive之间进行沟通

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_f995dc79e5f6bdd30e557b746106b100.jpg

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_576699a3cc9a128ace614631b779c30c.jpg

转换为Hfile
从MySQL导入SQOOP是import
从HDFS文件导出MySQL是导出
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_6f51e21f40216a23ec200699e755609b.jpg

delete-target-dir
//如果目录存在先删除
--target-dir /datas/sqoop/tb_user 
//输出文件到HDFS上指定目录
--fields-terminated-by '\t'
-m 1
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_1badb99d7d2ff98cebfb0a96041e9728.jpg

执行完毕

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_dcb2f6cb838a901e2c02d8fc2804e7d6.jpg

https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_1252f2e940ecd2b36ac129e16273a43d.jpg
https://yczbest.cn/wp-content/uploads/2020/08/wp_editor_md_9547b54b8e28dea311dc62cfed2abe48.jpg

发表评论

textsms
account_circle
email

大白糖奶兔的Blog

初识Hadoop-SQOOP(Day13)
前戏 配置运行SQOOP 配置 中间格式:Json tar -zxf sqoop-1.4.6-cdh5.7.6.tar.gz -C /opt/modules/ 目录结构 赋值Jar包 [hadoop@hadoop sqoop-1.4.6-cdh5.7.6]$ cp /o…
扫描二维码继续阅读
2020-08-05
Title - Artist
0:00