本文介紹SQL與Catalog報錯相關問題。

MySQL Catalog加載很慢或者加載出來的表名為灰色如何解決？

  • 報錯詳情
    MySQL Catalog創建成功但加載很慢或者加載出來的表名為灰色，注冊Catalog失敗，如圖所示。問題實例
  • 報錯原因

    從云產品到用戶MySQL的網絡特別卡，建立的連接經常斷掉，造成了不穩定現象。

  • 解決方案
    以下三種優化方式可供您選擇：
    • 您需要判斷云產品到MySQL走的公網帶寬是否受限（例如有些配置是默認非常小的2 MB），以及是否存在跨Region訪問的問題，詳情可參見如何查看公網帶寬情況？。
    • 如果您的網絡確實不佳，您可以將MySQL實例相關的interactive_timeout和wait_timeout參數的取值調大點。
    • 創建時您可以添加WITH參數'connect.timeout'='120s'。

報錯：Cannot obtain STS token from EMR meta-service.

  • 報錯詳情
    Caused by: MetaException(message:Initialize DlfMetaStoreClient failed: Initialize DlfMetaStoreClient failed: Cannot obtain STS token from EMR meta-service. Note that AK-Mode[dlf.catalog.akMode] can only used in EMR clusters, otherwise you should config the dlf.catalog.accessKeyId and dlf.catalog.accessKeySecret explicitly.)
  at com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient.createClient(ProxyMetaStoreClient.java:91)
  at com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient.<init>(ProxyMetaStoreClient.java:71)
  ... 41 more
  • 報錯原因

    配置文件有誤導致Hive Catalog初始化異常。

  • 解決方案
    1. 找到hive-conf-dir目錄下的hive-site.xml文件，刪除如下property信息。
       <property>
   <name>dlf.catalog.akMode</name>
   <value>EMR_AUTO</value>
 </property>
    2. 配置AccessKeyId和AccessKeySecret。
      <property>
    <name>dlf.catalog.accessKeyId</name>
    <value>${AccessKeyId}</value>
  </property>
<property>
    <name>dlf.catalog.accessKeySecret</name>
    <value>${AccessKeySecret}</value>
  </property>

報錯：RowTime field should not be null, please convert it to a non-null long value.

  • 報錯詳情
    2022-08-10 18:45:55,182 [flink-akka.actor.default-dispatcher-19] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - WatermarkAssigner(rowtime=[tsInMs], watermark=[WITHOFFSET(tsInMs, 0)]) with job vertex id 982161b1a590d7fd02ca6b6317c4**** (1/2) (2797311f5410de67f59aba510ae4****) switched from RUNNING to FAILED on job-82acab64-19c0-4e45-85cf-9bb49836****-taskmanager-1-1 @ 192.168.XX.XX (dataPort=39925).
java.lang.RuntimeException: RowTime field should not be null, please convert it to a non-null long value.
    at org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperator.processElement(WatermarkAssignerOperator.java:115) ~[flink-table-blink_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
    at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
    at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:135) ~[flink-dist_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
    at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:106) ~[flink-dist_2.11-1.13-vvr-4.0.14-2-SNAPSHOT.jar:1.13-vvr-4.0.14-2-SNAPSHOT]
  • 報錯原因

    生成Watermark的Event time時間字段中存在NULL值臟數據。

  • 解決方案
    使用計算列過濾臟數據，代碼示例如下。
    ts as case when `datetime` is null
           then to_timestamp('1970-01-01 00:00:00')
           else `datetime`end