之前用k3s搭建了大数据环境，但是发现不是特别方便，并且会反复修改，因此考虑到实验的便捷性又用docker搭了一下开发环境，所有的物料链接见https://github.com/henrywangx/dev-cluster
Dev cluster搭建
1.安装
前提：docker和docker-compose已经安装
1.拉起容器
make up
2.到 minio中国下载 下载mc客户端
3.添加dev集群到mc
mc config host add dev http://localhost:9000 minio minio123 --api s3v4
4.创建minio的accesskey/secret, 并保存到本地
5.使用minio的access key信息更新.env文件
# AWS_REGION is used by Spark
AWS_REGION=us-east-1
# This must match if using minio
MINIO_REGION=us-east-1
# Used by pyIceberg
AWS_DEFAULT_REGION=us-east-1
# AWS Credentials (this can use minio credential, to be filled in later)
AWS_ACCESS_KEY_ID=qUgyOn1f3rbQkXAgCYLa
AWS_SECRET_ACCESS_KEY=MJA9lmnlESWEJZgmJ5Itdee94DUF16wSMfyhsIzT
# If using Minio, this should be the API address of Minio Server
AWS_S3_ENDPOINT=http://minio:9000
# Location where files will be written when creating new tables
WAREHOUSE=s3a://openlake/
# URI of Nessie Catalog
NESSIE_URI=http://nessie:19120/api/v1
GRANT_SUDO=yes
重新拉起容器
make up
2.数据准备
1.创建bucket openlake/spark/sample-data/
# 输入bucket
mc mb dev/openlake/spark/sample-data/
# 输出bucket
mc mb dev/openlake-tmp/spark/nyc/taxis_small
2.下载出租车数据拷贝到minio中
wget https://data.cityofnewyork.us/api/views/t29m-gskq/rows.csv ./
mc cp rows.csv dev/openlake/spark/sample-data/
3.运行spark任务
1.网页访问jupyter地址: <localhost:8888>
2.运行spark-minio.py脚本
python3 spark-minio.py
3.查看spark管理页面:<localhost:8080>和jupyter的4040端口<localhost:4040>，分别可以查看运行的application信息和job的详情信息
application信息：
job详情:
4.等待python执行完毕，查看结果，可以看到外面算出来超过6名乘客的taxi为898
jovyan@jupyter-lab:~$ python3 spark-minio.py
Setting default log level to "WARN".
...
2024-01-28 07:55:20,121 - MinIOSparkJob - INFO - Total Rows for NYC Taxi Data: 91704300
2024-01-28 07:55:20,121 - MinIOSparkJob - INFO - Total Rows for Passenger Count > 6: 898
4.使用pyspark-iceberg管理table
1.创建warehouse bucket
mc mb dev/warehouse
2.运行spark-iceberg-minio.py
python3 spark-iceberg-minio.py
5.配置dremio
1.登录dremio页面：<localhost:9047>，创建s3的source
s3 source配置：
s3 advanced配置：
s3配置这里加了以下配置：
-
fs.s3a.path.style.access:true -
fs.s3a.endpoint:http://minio:9000 -
dremio.s3.compat:true - 勾选enable compatibility mode, 因为我们是minio
2.format table为iceberg，进入到nyc.taxis_large这个目录，然后点击format table的按钮，保存为iceberg
3.format为iceberg后，我们就能发现一个table，选中table，运行sql，发现我们可以用sql来操作iceberg表了，哈哈
SELECT * FROM taxis_large limit 10
参考
https://www.cnblogs.com/rongfengliang/p/17970071
https://github.com/minio/openlake/tree/main
https://www.linkedin.com/pulse/creating-local-data-lakehouse-using-alex-merced/
https://medium.com/@ongxuanhong/dataops-02-spawn-up-apache-spark-infrastructure-by-using-docker-fec518698993
https://medium.com/@ongxuanhong/are-you-looking-for-a-powerful-way-to-streamline-your-data-analytics-pipeline-and-produce-cc13ea326790
