Build a Github Project Summarization App

This tutorial demonstrates how to build a github project summerization app by BabelGPT. This app takes a project URL, retrieves related information and calls OpenAI model to do the summerization.

Requirements

User can input a github project url in the webpage.
When user click submit button, the webpage sends two requests to backend.
The first request for the following info:

a. Project name

b. Project description

c. Project star count

d. Project fork count

e. Project watch count
The second request (response in stream) for the following info:

a. Summarization of project readme
The webpage should show loading when waiting for a response.
The webpage should display the response under the url text box.
Summarization should be done by OpenAI model gpt-3.5-turbo-16k-0613

a. Prompt: Please summarize the README file of project {{project_name}}: \n {{readme_content}}

b. If readme_content is over 25000 characters, truncate it.

Analysis

This requirement only needs one webpage. The complicated part is at backend. We should use axios to call github api to get related data. Use LLMExecutor to call OpenAI with given prompt.

Building the App

Note: As we will generate the full app by BabelGPT, the result may differ from time to time. Also, the accuracy of AI's response is not guaranteed, you should inspect the result carefully.

Create app by BabelGPT and paste the above requirements.
You should see BabelGPT working on element structure and code. Please wait and do NOT operate in the workspace until BabelGPT finish the work.
Select element "callOpenAI" and click the settings button, to make sure the OpenAI settings are correct.
Switch to Edit View and select both callOpenAI element and Config, input your OpenAI API key.
If you don't have an OpenAI API key, you can use built-in provider. Please note that the built-in provider is for testing purpose only.
Submit your app
Click "Open", select index.html and click the "Open in new tab" button

You should see a page like this, input any GitHub project url such as "https://github.com/babelcloud/LLM-RGB" and click "Submit".

In this step, you may encounter all kinds of errors as BabelGPT may generate all kinds of code. You may use tools like browser console, Observation View to debug. I will paste a version of code that works correctly for your reference.

// HTTP element POST /projectInfo
import * as Koa from "koa";
import axios from "axios";

/**
 * API endpoint, handles HTTP POST "/projectInfo". Get the project url from POST message. Call Github API to get project info and return the info.
 */
export default async function(request: Koa.Request, response: Koa.Response, ctx: Koa.Context) {
    const projectUrl = request.body.url;
    const repository = projectUrl.split("/").slice(3).join("/");
    const apiUrl = `https://api.github.com/repos/${repository}`;
    const headers = {
        "Accept": "application/vnd.github.v3+json"
    };
    const result = await axios.get(apiUrl, { headers });
    const data = result.data;
    const projectInfo = {
        name: data.name,
        description: data.description,
        stars: data.stargazers_count,
        forks: data.forks_count,
        watches: data.watchers_count
    };
    return projectInfo;
}

// HTTP element POST /summarizeReadme
import * as Koa from "koa"
import axios from "axios";
import { truncateText, callOpenAI } from "#elements";

/**
 * API endpoint, handles HTTP POST "/summarizeReadme". Get the project url from POST message. Call Github API to get project readme. Truncate the readme if it's over 25000 characters. Call OpenAI to summarize the readme and return the summary.
 */
export default async function(request: Koa.Request, response: Koa.Response, ctx: Koa.Context) {
    const projectUrl = request.body.url;
    const repository = projectUrl.split("/").slice(3).join("/");
    const apiUrl = `https://api.github.com/repos/${repository}/readme`;
    const headers = {
        "Accept": "application/vnd.github.v3+json"
    };
    const result = await axios.get(apiUrl, { headers });
    let readmeContent = Buffer.from(result.data.content, 'base64').toString();
    if (readmeContent.length > 25000) {
        readmeContent = await truncateText(readmeContent, 25000);
    }
    const prompt = `Please summarize the README file of project ${projectUrl}: ${readmeContent}`;
    const summaryStream = await callOpenAI.completeStream(prompt);
    response.body = summaryStream.pipeThrough(new TransformStream({
        transform(chunk, controller) {
            controller.enqueue(chunk.choices[0]?.delta.content)
        }
    }));
    response.status = 200;
}

// Function element truncateText
import { TextEncoder } from "util";

/**
 * Truncate the given text if it's over the given length.
 * @param text - The text to be truncated.
 * @param length - The maximum length of the text.
 **/
export default async function(text: string, length: number) {
    const encoder = new TextEncoder();
    const encoded = encoder.encode(text);
    if (encoded.length <= length) {
        return text;
    }
    return encoder.decode(encoded.subarray(0, length));
}

Remember to add "axios" in Dependency. Here is the frontend code:

<!DOCTYPE html> <html> <head>
    <title>Github Project Info Retrieval and Summarization</title>

</head> <body>
    <h1>Github Project Info Retrieval and Summarization</h1>
    <form id="form">
        <label for="url">Github Project URL:</label>
        <input type="text" id="url" name="url">
        <button type="submit">Submit</button>
    </form>
    <div id="loading" style="display: none;">Loading...</div>
    <div id="info"></div>
    <div id="summary"></div>
    <script src="/main.js"></script>
</body> </html>

document.getElementById('form').addEventListener('submit', function(e) {
    e.preventDefault();
    const url = document.getElementById('url').value;
    document.getElementById('loading').style.display = 'block';
    fetch('/projectInfo', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ url: url })
    })
    .then(response => response.json())
    .then(data => {
        document.getElementById('info').innerHTML = `
            <h2>Project Info</h2>
            <p>Name: ${data.name}</p>
            <p>Description: ${data.description}</p>
            <p>Stars: ${data.stars}</p>
            <p>Forks: ${data.forks}</p>
            <p>Watches: ${data.watches}</p>
        `;
        return fetch('/summarizeReadme', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({ url: url })
        });
    })
    .then(response => response.text())
    .then(data => {
        document.getElementById('summary').innerHTML = `
            <h2>Readme Summary</h2>
            <p>${data}</p>
        `;
        document.getElementById('loading').style.display = 'none';
    });
});

If your code runs correctly, you should see the page behaves like this:

Now, let's try to make the summary show up word by word. Open assets item "main.js", and hit "cmd+b".

Input instruction "The /summarizeReadme API returns a stream. Please modify the code to process the stream so that the response shows up word by word." and click "Send". You should see BabelGPT revised the code and shows the diff

Click "Accept" and Submit the app again. Follow instruction in step 9 and you should see the summary comes out word by word. Please note the BabelGPT may not revise "main.js" correctly. Here is the a working version of "main.js":

document.getElementById('form').addEventListener('submit', function(e) {
    e.preventDefault();
    const url = document.getElementById('url').value;
    document.getElementById('loading').style.display = 'block';
    fetch('/projectInfo', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({ url: url })
    })
    .then(response => response.json())
    .then(data => {
        document.getElementById('info').innerHTML = `
            <h2>Project Info</h2>
            <p>Name: ${data.name}</p>
            <p>Description: ${data.description}</p>
            <p>Stars: ${data.stars}</p>
            <p>Forks: ${data.forks}</p>
            <p>Watches: ${data.watches}</p>
        `;
        return fetch('/summarizeReadme', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json'
            },
            body: JSON.stringify({ url: url })
        });
    })
    .then(response => {
        const reader = response.body.getReader();
        const decoder = new TextDecoder('utf-8');
        let buffer = '';
        function processStream() {
            return reader.read().then(({ done, value }) => {
                buffer += decoder.decode(value, { stream: !done });
                let boundary = buffer.lastIndexOf(' ');
                if (boundary !== -1) {
                    let piece = buffer.slice(0, boundary);
                    buffer = buffer.slice(boundary + 1);
                    document.getElementById('summary').innerHTML += piece + ' ';
                }
                if (done) {
                    document.getElementById('loading').style.display = 'none';
                    return;
                }
                return processStream();
            });
        }
        return processStream();
    });
});

Build a Github Project Summarization App

Requirements​

Analysis​

Building the App​

Requirements

Analysis

Building the App