김민서AZURE CLOUD AI ENGINEERv0.2.6

TECHNICAL KNOWLEDGE BASE

Azure VM Custom Log

VM Log Analytics Workspace Custom Log Table 인프라 구성 절차 1. Log Analytics Workspace 생성 2. Data Collection Endpoint DCE 생성 3. Data Collection Rule DCR 생성

Azure Monitor
Azure

Legacy Engineering Note · 기존 기술 블로그에서 선별 이관한 기록입니다. 작성 당시 환경과 버전을 기준으로 합니다.

VM Log Analytics Workspace Custom Log Table

인프라 구성 절차

  1. Log Analytics Workspace 생성

  2. Data Collection Endpoint(DCE) 생성

  3. Data Collection Rule(DCR) 생성

    • 별도로 생성하지 않음
    • 생성하기 위해 Log Analytics Workspace의 Custom Table이 이미 생성되어야함
    • DCR를 별도로 생성하지 않고 Log Analytics Workspace에서 Custom Log table생성시 같이 생성

  4. Custom Log table 생성

    • 생성과정
      1. Log Analytics workspace > Setting > Tables
      2. Create : New custom log(DCR-based)
    1. Basics
      • Data collection rule : 생성시 DCR이 없으면 DCR 생성
      • Data collection endpoint : 생성한 DCE 지정
    2. Schema and transformation
    json
    [
    {
    "Time": "2023-03-12T15:04:48.423211Z",
    "Computer": "Computer1",
        "AdditionalContext": {
            "InstanceName": "user1",
            "TimeZone": "Pacific Time",
            "Level": 4,
            "CounterName": "AppMetric2",
            "CounterValue": 35.3    
        }
    },
    {
        "Time": "2023-03-12T15:04:48.794972Z",
        "Computer": "Computer2",
        "AdditionalContext": {
            "InstanceName": "user2",
            "TimeZone": "Central Time",
            "Level": 3,
            "CounterName": "AppMetric2",
            "CounterValue": 43.5     
        }
    }
]
    • [Error] : TimeGenerated field is not found in the sample provided. Please use the transformation editor to populate TimeGenerated column in the destination table.
      1. </>Transformatio editor 클릭
      2. 아래 쿼리 실행 후 Apply눌러 적용
    kql
    source
| extend TimeGenerated = todatetime(Time)

  5. 권한 설정[확인 예정]

  6. Python Code 전송 테스트

bash
pip install azure-monitor-ingestion
pip install azure-identity
python
# information needed to send data to the DCR endpoint
endpoint_uri = "https://my-url.monitor.azure.com" # logs ingestion endpoint of the DCR
dcr_immutableid = "dcr-00000000000000000000000000000000" # immutableId property of the Data Collection Rule

# [중요!]custom table명이 아님
# 생성된 DCR 개요 > Json View 클릭 > 생성한 "custom table명"으로 검색 "Custom-Text-"으로 시작
stream_name = "Custom-MyTableRawData" #name of the stream in the DCR that represents the destination table

# Import required modules
import os
from azure.identity import DefaultAzureCredential
from azure.monitor.ingestion import LogsIngestionClient
from azure.core.exceptions import HttpResponseError

credential = DefaultAzureCredential()
client = LogsIngestionClient(endpoint=endpoint_uri, credential=credential, logging_enable=True)

body = [
        {
        "Time": "2023-03-12T15:04:48.423211Z",
        "Computer": "Computer1",
            "AdditionalContext": {
                "InstanceName": "user1",
                "TimeZone": "Pacific Time",
                "Level": 4,
                "CounterName": "AppMetric2",
                "CounterValue": 35.3    
            }
        },
        {
            "Time": "2023-03-12T15:04:48.794972Z",
            "Computer": "Computer2",
            "AdditionalContext": {
                "InstanceName": "user2",
                "TimeZone": "Central Time",
                "Level": 3,
                "CounterName": "AppMetric2",
                "CounterValue": 43.5     
            }
        }
    ]

try:
    client.upload(rule_id=dcr_immutableid, stream_name=stream_name, logs=body)
except HttpResponseError as e:
    print(f"Upload failed: {e}")
python
# information needed to send data to the DCR endpoint
endpoint_uri = "https://my-url.monitor.azure.com" # logs ingestion endpoint of the DCR
dcr_immutableid = "dcr-00000000000000000000000000000000" # immutableId property of the Data Collection Rule

# [중요!]custom table명이 아님
# 생성된 DCR 개요 > Json View 클릭 > 생성한 "custom table명"으로 검색 "Custom-Text-"으로 시작
stream_name = "Custom-MyTableRawData" #name of the stream in the DCR that represents the destination table

# Import required modules
import os
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.monitor.ingestion import LogsIngestionClient
from azure.core.exceptions import HttpResponseError

# credential = DefaultAzureCredential()
credential = ManagedIdentityCredential()
client = LogsIngestionClient(endpoint=endpoint_uri, credential=credential, logging_enable=True)

body = [
        {
        "Time": "2023-03-12T15:04:48.423211Z",
        "Computer": "Computer1",
            "AdditionalContext": {
                "InstanceName": "user1",
                "TimeZone": "Pacific Time",
                "Level": 4,
                "CounterName": "AppMetric2",
                "CounterValue": 35.3    
            }
        },
        {
            "Time": "2023-03-12T15:04:48.794972Z",
            "Computer": "Computer2",
            "AdditionalContext": {
                "InstanceName": "user2",
                "TimeZone": "Central Time",
                "Level": 3,
                "CounterName": "AppMetric2",
                "CounterValue": 43.5     
            }
        }
    ]

try:
    client.upload(rule_id=dcr_immutableid, stream_name=stream_name, logs=body)
except HttpResponseError as e:
    print(f"Upload failed: {e}")

참고 문서